Nice post regarding boot from EBS.

On Error: fsck died with exit status 8

I think one possible cause of this is when fsck found inconsistent block on EBS volume used.

A possible solution could be ignoring filesystem check by editing the fstab setting pass to 0. As posted at http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24091&start;=30&tstart;=0

Or either attached that EBS volume to another instance and do fsck to that device then reattached it again to the pivot AMI instance.

Just a thought.

Rodney

Yes, there are many cool use cases enabled by shared snapshots. I even wrote an article about some: http://www.shlomoswidler.com/2009/09/cool-things-you-can-do-with-shared-ebs.html

I wish I could claim credit for "getting through to them" but I shouldn't. It takes more than one person to clamor, and AWS seem to be strongly influenced by customer clamor.

It's true – any distro can be used to build the boot AMI, and so the public AMI you mentioned (ami-2feb0f46 nimlabs/pivot-sdj-20080824.manifest.xml) can be used to boot from any EBS attached to /dev/sdj . Anyone who needs a boot AMI using a different attach point can follow the procedure in this article to build one from an Ubuntu AMI.

I did identify that the difference between Ubuntu and Debian lies in the RAID support. As you say, there appears to be little practical value to discovering the underlying cause of failure.

Thanks for your feedback!

I hate to think that my brand loyalty has got you delving into the differences between Debian and Ubuntu or Fedora, when any distro will suffice for the pivot AMI.

Yes, it turns out that the actual problems begin later in the boot sequence. The entire console output (which can take a while to show up, and sometimes only flushes completely at instance termination) shows more interesting stuff.

Boot-time problems are notoriously hard to debug in EC2 where you can't get console access.

Using the same AMI/AKI/ARI as you, I get this:
[skipping uninteresting stuff]
EXT3 FS on sda1, internal journal
Making pivot_root device node /new-root-dev
Attempting to mount pivot_root volume on /new-root-dev
1 scsi_mod: no version for "struct_module" found: kernel tainted.
SCSI subsystem initialized
register_blkdev: cannot get major 8 for sd
mount: /new-root-dev is not a valid block device
2 mount: /new-root-dev is not a valid block device
3 mount: /new-root-dev is not a valid block device
4 mount: /new-root-dev is not a valid block device
5 mount: /new-root-dev is not a valid block device
6 mount: /new-root-dev is not a valid block device
7 mount: /new-root-dev is not a valid block device
8 mount: /new-root-dev is not a valid block device
9 mount: /new-root-dev is not a valid block device
10 mount: /new-root-dev is not a valid block device
11 mount: /new-root-dev is not a valid block device
12 mount: /new-root-dev is not a valid block device
13 mount: /new-root-dev is not a valid block device
14 mount: /new-root-dev is not a valid block device
15 mount: /new-root-dev is not a valid block device
16 mount: /new-root-dev is not a valid block device
17 mount: /new-root-dev is not a valid block device
18 mount: /new-root-dev is not a valid block device
sdp: unknown partition table
19 kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Mounted pivot_root volume.
Mount failed for selinuxfs on /selinux: No such file or directory
INIT: version 2.86 booting
/proc is empty (not mounted ?)
hostname: the specified hostname is invalid
Starting the hotplug events dispatcher: udevd.
Synthesizing the initial hotplug events…done.
Waiting for /dev to be fully populated…input: PC Speaker as /class/input/input1
done.
Activating swap…Adding 917496k swap on /dev/sda3. Priority:-1 extents:1 across:917496k
done.
Checking root file system…fsck 1.41.3 (12-Oct-2008)
fsck.ext3: Device or resource busy while trying to open /dev/sdp
Filesystem mounted or opened exclusively by another program?
fsck died with exit status 8
[31mfailed (code 8).[39;49m
An automatic file system check (fsck) of the root filesystem failed. A manual fsck must be performed, then the system restarted. The fsck should be performed in maintenance mode with the root filesystem mounted in read-only mode. [31mfailed![39;49m
The root filesystem is currently mounted in read-only mode. A maintenance shell will now be started. After performing system maintenance, press CONTROL-D to terminate the maintenance shell and restart the system. [33m(warning).[39;49m
Press enter for maintenance
(or type Control-D to continue):

The 19 retries happened while waiting for the volume to be attached to /dev/sdp. Then the boot continues OK but dies with:
Checking root file system…fsck 1.41.3 (12-Oct-2008)
fsck.ext3: Device or resource busy while trying to open /dev/sdp
Filesystem mounted or opened exclusively by another program?
fsck died with exit status 8

Ugh. I'll look into this some more.

Checking root file system…fsck 1.41.3 (12-Oct-2008)
fsck.ext3: Device or resource busy while trying to open /lib/init/rw/rootdev
Filesystem mounted or opened exclusively by another program?
fsck died with exit status 8

In 20 or so boots I found that the public pivot AMI also failed once, so the problem may be a subtle one.

However… I made a happy discovery while I was playing with this, which probably seems obvious to you, but which may interest your readers.

I (shortsightedly?) created a 1G EBS volume for my DEV environment, which was OK for experimenting but very soon filled up. I wished I made a 2G volume.

I thought of going back to the drawing board, but then it occurred to me that I could do this:

1. Create a snapshot of the 1G bootable EBS volume with my DEV system on it.
2. Create a 2G volume from that snapshot.
3. Attach the new volume as (say) /dev/sdk, but don't mount it.
4. Resize the filesystem to 2G
e2fsck -f /dev/sdk
resize2fs /dev/sdk
6. Snapshot the volume with the resized filesystem
7. Use my new snapshot henceforth as the bootable EBS volume.

This quickly gives me a way to increase the size of the bootable volume. An elastic smile appeared on my face

The problem reported by scsi_mod may be an issue with the kernel modules included in the ramdisk. What AKI and ARI did you use? I'll try to reproduce with the default AKI and ARI and see what happens….

The ability to create public, shared EBS volumes has been requested by many. It would allow some really cool sharing scenarios, such as the one you describe. I have submitted a question to the AWS folks who manage the public data set submissions:

—
I'd like to share an EBS volume with the public in the same way I share an AMI. Public EBS snapshots would allow many really cool use cases, including bypassing the 10GB root-partition limit of an AMI (in combination with the pivot_root technique). Can I submit an EBS snapshot containing the linux boot code and startup sequence? Would you approve it? It's more than "data", it would also be the kind of stuff you'd burn into an AMI – apps, configs, etc….
—

I'll report back what they say.