[SAC] ProjectsVM Upgrade Problem

Hi OSU OSL Admins,

We have attempted a Debian Dist upgrade today for the
OSGeo ProjectsVM (projects.osgeo.osuosl.org). While it
mostly seems to have gone well, after reboot it did not come
back properly. We suspect grub issues. Lacking console
access it is hard for us to diagnose.

Could someone take a look and/or provide us with console
access? I shall endevour to be available (FrankW) in #osuosl
to consult.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up | Frank Warmerdam, warmerdam@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush | Geospatial Software Developer

On 02/01/2012 05:25 PM, Frank Warmerdam wrote:

Hi OSU OSL Admins,

We have attempted a Debian Dist upgrade today for the
OSGeo ProjectsVM (projects.osgeo.osuosl.org). While it
mostly seems to have gone well, after reboot it did not come
back properly. We suspect grub issues. Lacking console
access it is hard for us to diagnose.

Could someone take a look and/or provide us with console
access? I shall endevour to be available (FrankW) in #osuosl
to consult.

Best regards,

Lance looked into it and fixed grub. Here are some notes/suggestions for
the next machine we decide to upgrade:

run grub-install --force /dev/vda before rebooting

Has something to do with how our virtual machine disks were originally
made and the fact that the debian upgrade is pushing grub2 as a
replacement to grub.

ProjectsVM should now be booted. Looks like it's ready for login, is the
upgrade done?

Thanks,
Alex

Hi,
even at the risk of people telling me again, that I'm wrong, I'd like
to advertize (again) the use of the 'traditional' Debian dist-upgrade
method, which is:

Edit /etc/apt/sources.list

#~ > aptitude update
#~ > aptitude install dpkg apt aptitude # probably add locales-all to
                                         # avoid some of the boring
                                         # perl-warnings
#~ > aptitude dist-upgrade -d # optional but nice
#~ > aptitude dist-upgrade # Yes to removing the running
                                         # kernel

This way I've successfully dist-upgraded more than a dozend ! Debian
systems from Lenny to Squeeze, including Xen-, VMware- and two of the
OSGeo KVM-VM's. Just a single one, a 'physical' machine ran into
trouble after reboot because I forgot to update the custom kernel
before dist-upgrading and, as a consequence, udev was unable to
properly create the devices.

Cheers,
  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

On Thu, Feb 2, 2012 at 5:36 AM, Alex Mandel <tech_dev@wildintellect.com> wrote:
...

Lance looked into it and fixed grub.

Great.

ProjectsVM should now be booted. Looks like it's ready for login, is the
upgrade done?

Yes, should be fine, I have done all steps.

Remaining:

In daemon.log I get:

Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: 0 processes alive
and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping'
resulted in
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:
#007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: error: 'Can't
connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)'
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: Check that mysqld is
running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:

No idea how to debug this :frowning:

Markus

Martin Spott wrote:

even at the risk of people telling me again, that I'm wrong,
I'd like to advertize (again) the use of the 'traditional' Debian
dist-upgrade method,

times change, methods change, stick with what works for you, but always
spend a few minutes glancing at the release notes...

which is:

Edit /etc/apt/sources.list

#~ > aptitude update

the release notes this time tell us that aptitude is no longer the
favoured method, and apt-get is back in fashion. if we wait long enough
maybe they'll start recommending dselect again. :slight_smile:

anyway the troubles so far hasn't been to do with the order or method
of package upgrades leading to a deadlocked dpkg database, and the VMs
need to be reboot-proof*, so whatever method and whatever apt frontend
is chosen shouldn't really be an issue. neither one's "wrong", as many
roads lead to rome.

[*] n.b. adhoc currently wants a reboot due to a new kernel security
update, but the grub package was installed since the last boot, and so
there's a good chance it will run into the same reboot problem which the
projects VM had. what needs to be done?

* fsck problems
  - if fsck can't be run on a mounted (ie root) fs try for reboot just
    before starting the upgrade?

* grub2 problems
  - see if it is possible to apply whatever change was needed to fix
    the projects VM to all the rest of the VMs' /dev/vda just before
    the fsck reboot suggested above.

regards,
Hamish

Markus wrote:

Remaining:

In daemon.log I get:

Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: 0
processes alive
and '/usr/bin/mysqladmin
--defaults-file=/etc/mysql/debian.cnf ping'
resulted in
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:
#007/usr/bin/mysqladmin: connect to server at 'localhost'
failed
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:
error: 'Can't
connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)'
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: Check
that mysqld is
running and that the socket: '/var/run/mysqld/mysqld.sock'
exists!
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:

No idea how to debug this :frowning:

try
  sudo /etc/init.d/mysql restart

and see what happens. error messages may be getting masked, so ensure
that /etc/default/rcS has "VERBOSE=yes".

Hamish

On Thu, Feb 02, 2012 at 12:23:48AM -0800, Hamish wrote:

  - see if it is possible to apply whatever change was needed to fix
    the projects VM to all the rest of the VMs' /dev/vda just before
    the fsck reboot suggested above.

Please don't on the "backup" VM,

  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------

Hamish wrote:

try
  sudo /etc/init.d/mysql restart

and see what happens. error messages may be getting masked,
so ensure that /etc/default/rcS has "VERBOSE=yes".

actually, try first:

sudo /etc/init.d/mysql-ndb-mgm restart
sudo /etc/init.d/mysql-ndb restart

Hamish

On Thu, Feb 2, 2012 at 9:30 AM, Hamish <hamish_b@yahoo.com> wrote:

Markus wrote:

Remaining:

In daemon.log I get:

Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: 0
processes alive
and '/usr/bin/mysqladmin
--defaults-file=/etc/mysql/debian.cnf ping'
resulted in
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:
#007/usr/bin/mysqladmin: connect to server at 'localhost'
failed
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:
error: 'Can't
connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock' (2)'
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]: Check
that mysqld is
running and that the socket: '/var/run/mysqld/mysqld.sock'
exists!
Feb 1 23:52:37 projects /etc/init.d/mysql[5514]:

No idea how to debug this :frowning:

try
sudo /etc/init.d/mysql restart

Yes, that's what I tried...

and see what happens. error messages may be getting masked, so ensure
that /etc/default/rcS has "VERBOSE=yes".

projects:/home/neteler# /etc/init.d/mysql start
Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!
projects:/home/neteler# grep yes /etc/default/rcS
UTC=yes
VERBOSE=yes

Not verbose at all :frowning:

Markus

On Thu, Feb 2, 2012 at 9:38 AM, Hamish <hamish_b@yahoo.com> wrote:

Hamish wrote:

try
sudo /etc/init.d/mysql restart

and see what happens. error messages may be getting masked,
so ensure that /etc/default/rcS has "VERBOSE=yes".

actually, try first:

sudo /etc/init.d/mysql-ndb-mgm restart
sudo /etc/init.d/mysql-ndb restart

projects:/home/neteler# /etc/init.d/mysql-ndb-mgm restart
projects:/home/neteler# /etc/init.d/mysql-ndb restart
projects:/home/neteler# /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld . . . . . . . . . . . . . . failed!

Markus

On Thu, Feb 02, 2012 at 12:23:48AM -0800, Hamish wrote:

Martin Spott wrote:

> even at the risk of people telling me again, that I'm wrong,
> I'd like to advertize (again) the use of the 'traditional' Debian
> dist-upgrade method,

times change, methods change, stick with what works for you, but always
spend a few minutes glancing at the release notes...

Did anyone allege that I didn't ? :wink:

anyway the troubles so far hasn't been to do with the order or method
of package upgrades leading to a deadlocked dpkg database,

Ah ?

  Martin.
--
Unix _IS_ user friendly - it's just selective about who its friends are !
--------------------------------------------------------------------------