Hola all,
So this is my first blog post since UDS started (although I have been doing some work microblogging this session around (I find that if I treat it like multicast IRC using gwibber, it suddenly makes more sense to me). We're now three days into UDS, and working hard on defining what Ubuntu karmic will be, and I must say I am excited with the way things are shaping up to UNR discussions, to the Android Execution Environment (and if anyone has any questions on it, please direct those emails to Michael Frey and Debbie Beliveau as they are the people behind it, despite Slashdot's reports on the subject).
There are loads going on, including Moblin (which you'll see this afternoon), Android (same), ports kernel handling, and loads of other cool things come up. I'll comment on some of the more interesting things as time goes on.
In other news, as of late last night, I'm officially an upstream KDE developer with SVN commit writes. I've written an email detailing my plans for working on KDE to kde-core-deveonl, where is it is happily stuck in a moderation queue, so hopefully those involved in upstream KDE development will soon learn of my intentions :-).
I'll write more later,
Michael
Wednesday, May 27, 2009
Friday, April 24, 2009
Jaunty Retrospect
Guess its my time to post something about my feelings on Jaunty.
The Good:
* armel successfully birthed
* powerpc well on the way to be well maintained
- Kernel and installer work mostly done by TheMuso (thanks :-))
- Image testing by the people of #ubuntu-ps3, myself, and TheMuso
- powerpc, and powerpc+ps3 both in the release annoucements for Kubuntu and Xubuntu :-)
* Kubuntu upgraded to KDE 4.2
* Xubuntu upgraded to Xfce 4.6
* PowerPC FTBFS rate in main very low. ia64, and sparc looking more improved.
The Bad:
* SPARC, ia64, and HPPA remain fairly foobar w.r.t. to the installer and kernel
The Ugly:
* The drama over notifications and update-manager
All and all though, I think its been a fairly good cycle. Looking forward to karmic.
The Good:
* armel successfully birthed
* powerpc well on the way to be well maintained
- Kernel and installer work mostly done by TheMuso (thanks :-))
- Image testing by the people of #ubuntu-ps3, myself, and TheMuso
- powerpc, and powerpc+ps3 both in the release annoucements for Kubuntu and Xubuntu :-)
* Kubuntu upgraded to KDE 4.2
* Xubuntu upgraded to Xfce 4.6
* PowerPC FTBFS rate in main very low. ia64, and sparc looking more improved.
The Bad:
* SPARC, ia64, and HPPA remain fairly foobar w.r.t. to the installer and kernel
The Ugly:
* The drama over notifications and update-manager
All and all though, I think its been a fairly good cycle. Looking forward to karmic.
Thursday, January 15, 2009
Re-enginneering my network ...
As I move to having more and more machines on my internal LAN, I felt the time had finally come that I sit down, and rebuild my network to take advantage of things such as gigabyte networking, LDAP, single-user sign on, and so forth. I'm doing partially for fun, and partially because its an interesting experiment to see how Linux from an IS environment compares to a Windows 200x IS environment (one of my former jobs was a 2000/XP/2003/Vista sysadmin position).
So, here's my current network setup
blacksteel <- *wireless* ---------------------------------------------- cerberus <-> Internet
/
dawn <------ *wired*-----------------------------------------------------
/
360 <---- *wired* -----------------------------------------------------
Online machines:
cerberus - WRT51GS
backsteel - My laptop
dawn - Development machine
360 - Xbox 360, used to play media from blacksteel
Offline machines (aka, machines I have, but haven't fired up since moving:
helios (PowerMac G4)
apollo (old Dell P3)
junker (RS/6000 rescued from the dumpster, might be dead)
alexandria (NSLU2; gave up its plug for dawn)
coldfusion (Coldfire Board, might be dead; ethernet controller is faulty, but might be able to use a USB based one to breath some life into it; can't autoreboot due to built in bootloader not supporting it; and no JTAG to sanely change the default bootloader).
siren (old MacBook Pro, has a dead internal HDD, but runs fine from an external hard drive. Was my Debian test box until its HDD went to dawn)
exodius - second WRT54GS used to be part of a WDS bridge.
unnamed dev box (not here yet, but likely soon).
Of all these machines, only apollo has a wireless card which ATM is non-functional. In addition, the wired bits of my network are 100Mbps, with a g based wireless hotspot (WPA secured). Futhermore, blacksteel, helios, and siren have gigabyte ethernet. apollo has 100MBps ethernet card. alexandria and dawn have 10MBps, which is painful, especially for NFS root.
I'll drop another 1Gbps NIC into apollo, replacing its wireless card, and give dawn, alexandria, and maybe coldfusion USB based NICs once I get around to resurrecting systems (alexandria and coldfusion don't have hard drives at the moment)
What I would like to do is use an Linux-based router and replace Cerberus. Helios has two gigabyte NICs, so it will take up this duty, as well as provide DHCPv4, and radvd (for IPv6) for the internal network. It's an old computer, and has an onboard model, and its position in my apartment will be close to a phone jack; maybe I'll set it up so I can dial in from outside the LAN in case something goes down (although my phones here are VoIP based so I dunno how useful that's going to be :-)).
Another box (I might task this to apollo, or helios) will run LDAP and NFS services, providing both a netboot based installation with preseed for fast re-installation, and NFS home folders for all machines except blacksteel (unless someone knows a great solution for having a laptop sync NFS and local home folders. helios will run mail, news, and any other untrusted net facing services, with everything else shielded behind it. All machines will run IPv4 and 6.
Anyway, this is the start of my plan in a nutshell, and I intend to continue discussion as I slowly build and implement this updated setup. Wish me luck :-).
So, here's my current network setup
blacksteel <- *wireless* ---------------------------------------------- cerberus <-> Internet
/
dawn <------ *wired*-----------------------------------------------------
/
360 <---- *wired* -----------------------------------------------------
Online machines:
cerberus - WRT51GS
backsteel - My laptop
dawn - Development machine
360 - Xbox 360, used to play media from blacksteel
Offline machines (aka, machines I have, but haven't fired up since moving:
helios (PowerMac G4)
apollo (old Dell P3)
junker (RS/6000 rescued from the dumpster, might be dead)
alexandria (NSLU2; gave up its plug for dawn)
coldfusion (Coldfire Board, might be dead; ethernet controller is faulty, but might be able to use a USB based one to breath some life into it; can't autoreboot due to built in bootloader not supporting it; and no JTAG to sanely change the default bootloader).
siren (old MacBook Pro, has a dead internal HDD, but runs fine from an external hard drive. Was my Debian test box until its HDD went to dawn)
exodius - second WRT54GS used to be part of a WDS bridge.
unnamed dev box (not here yet, but likely soon).
Of all these machines, only apollo has a wireless card which ATM is non-functional. In addition, the wired bits of my network are 100Mbps, with a g based wireless hotspot (WPA secured). Futhermore, blacksteel, helios, and siren have gigabyte ethernet. apollo has 100MBps ethernet card. alexandria and dawn have 10MBps, which is painful, especially for NFS root.
I'll drop another 1Gbps NIC into apollo, replacing its wireless card, and give dawn, alexandria, and maybe coldfusion USB based NICs once I get around to resurrecting systems (alexandria and coldfusion don't have hard drives at the moment)
What I would like to do is use an Linux-based router and replace Cerberus. Helios has two gigabyte NICs, so it will take up this duty, as well as provide DHCPv4, and radvd (for IPv6) for the internal network. It's an old computer, and has an onboard model, and its position in my apartment will be close to a phone jack; maybe I'll set it up so I can dial in from outside the LAN in case something goes down (although my phones here are VoIP based so I dunno how useful that's going to be :-)).
Another box (I might task this to apollo, or helios) will run LDAP and NFS services, providing both a netboot based installation with preseed for fast re-installation, and NFS home folders for all machines except blacksteel (unless someone knows a great solution for having a laptop sync NFS and local home folders. helios will run mail, news, and any other untrusted net facing services, with everything else shielded behind it. All machines will run IPv4 and 6.
Anyway, this is the start of my plan in a nutshell, and I intend to continue discussion as I slowly build and implement this updated setup. Wish me luck :-).
Friday, January 2, 2009
Notes from Underground, Part 1
For those following d-devel, you may notice that I've recently been working on improving one of the cornerstones of Debian infrastructure; the Debian Archive Kit, or dak for short. Most DDs and DMs don't notice dak exists expect when trying to determine why their latest upload was rejected, and then yelling at the powers that be. I'm here to shead some light on this mythicial beast.
First off, a quick history lesson:
dak (also known as projectb) is a replacement for Debian's original archive software, known simply as dinstall. dinstall itself was a fairly large perl script that does what dak process-unchecked/process-accepted does today. James Troup did a fairly decent summary of dinstall, and its issues
James Troup's README.new-incoming (from dak's git repo):
dak's first commits were in 2000, and rolled out onto ftp-master.d.o sometime in 2001 or 2002 (I can't find an exact date for this). Since then, dak is also used on security.d.o, and on backports.org (fun fact for bpo people; the dak installation there is now up to date, and tracking git's tip).
So now that you know the history lesson, what specificially does dak do is the next question. Simply put, dak is the glue that binds the rest of the Debian's backends together; both britney and wanna-build/buildd depend on it. It handles management of uploads to the archive, handles stable release updates, as so forth. It is also the only Debian archive software that uses an actual database backend, and scales fairly well handling over 10,000 packages, and 12 architectures. Unfortunately, there are also a lot of issues with dak as it stands.
Sections of the code base have bitrotted over the years; legacy and legacy-mixed support have died, the import-archive function is shot (more so now than ever, see below), the test suite is non-functional (never a good sign), the docs are out of date, and in many places non-existant, doing a release (both point and full) requires editing the database and so forth.
In addition, dak, while written in python, is written in a fairly procedural style, and and some very ugly code in some places. For instance, the original Debian Maintainer code was handled by having the uid's in the database prefixed by dm: vs having a flag somewhere, and had some hardcoded variables like checking for "unstable", as well as quite a few bugs which caused interesting behavior when uploading to a non-unstable suite such as experimental or one of the proposed queues. (for those of curious, I recommend checking the dak git tree to see what the old DM code looked like, and then aside from the design, find the two major bugs which caused a lot of the weirdness with DMs). It should be stated that the last merge from redid the DM code and design sanely using the new update framework.
These issues have lead to the genesis of the dak v2 project, which is an attempt to replace dak with a module, rewritten from the ground up to be more secure and modular, although its not gotten very far as of writing. I personally don't believe that the current iteration of dak is so bad as scrapping and rewritting is necessary. Instead, I've been working to implement v2 features in dak by aggressive refactoring and cleanup, with the hope of negating the need for a rewrite.
So now thats out of the way, I bet you probably are interested in my .plan for dak. Well, lets go over I've implemented so far.
* An update database framework for dak, which will allow for easy database upgrade and migration, vs the "does it work yet?" approach to applying schema updates. Simply type dak update-db, and your done!
* 822 formatted output for queues (http://ftp-master.debian.org/new.822); this information is now used on DDPO pages
* Rewriting DM management code to have more of a brain than the previous implementation.
What's next on the TODO list
* Content file generation from the database (part of removal of apt-ftparchive, but thats another blog post ;-)).
Oh, as a side note to my current readers, my blog has changed names to "Notes from Underground", after one of my favorite novels, and futher in reference to exploring the mysterious underground that is Debian's backend code. We're also now on Planet Debian :-).
First off, a quick history lesson:
dak (also known as projectb) is a replacement for Debian's original archive software, known simply as dinstall. dinstall itself was a fairly large perl script that does what dak process-unchecked/process-accepted does today. James Troup did a fairly decent summary of dinstall, and its issues
James Troup's README.new-incoming (from dak's git repo):
The old system:
---------------
o incoming was a world writable directory
o incoming was available to everyone through http://incoming.debian.org/
o incoming was processed once a day by dinstall
o uploads in incoming had to have been there > 24 hours before they
were REJECTed. If they were processed before that and had
problems they were SKIPped (with no notification to the maintainer
and/or uploader).
dak's first commits were in 2000, and rolled out onto ftp-master.d.o sometime in 2001 or 2002 (I can't find an exact date for this). Since then, dak is also used on security.d.o, and on backports.org (fun fact for bpo people; the dak installation there is now up to date, and tracking git's tip).
So now that you know the history lesson, what specificially does dak do is the next question. Simply put, dak is the glue that binds the rest of the Debian's backends together; both britney and wanna-build/buildd depend on it. It handles management of uploads to the archive, handles stable release updates, as so forth. It is also the only Debian archive software that uses an actual database backend, and scales fairly well handling over 10,000 packages, and 12 architectures. Unfortunately, there are also a lot of issues with dak as it stands.
Sections of the code base have bitrotted over the years; legacy and legacy-mixed support have died, the import-archive function is shot (more so now than ever, see below), the test suite is non-functional (never a good sign), the docs are out of date, and in many places non-existant, doing a release (both point and full) requires editing the database and so forth.
In addition, dak, while written in python, is written in a fairly procedural style, and and some very ugly code in some places. For instance, the original Debian Maintainer code was handled by having the uid's in the database prefixed by dm: vs having a flag somewhere, and had some hardcoded variables like checking for "unstable", as well as quite a few bugs which caused interesting behavior when uploading to a non-unstable suite such as experimental or one of the proposed queues. (for those of curious, I recommend checking the dak git tree to see what the old DM code looked like, and then aside from the design, find the two major bugs which caused a lot of the weirdness with DMs). It should be stated that the last merge from redid the DM code and design sanely using the new update framework.
These issues have lead to the genesis of the dak v2 project, which is an attempt to replace dak with a module, rewritten from the ground up to be more secure and modular, although its not gotten very far as of writing. I personally don't believe that the current iteration of dak is so bad as scrapping and rewritting is necessary. Instead, I've been working to implement v2 features in dak by aggressive refactoring and cleanup, with the hope of negating the need for a rewrite.
So now thats out of the way, I bet you probably are interested in my .plan for dak. Well, lets go over I've implemented so far.
* An update database framework for dak, which will allow for easy database upgrade and migration, vs the "does it work yet?" approach to applying schema updates. Simply type dak update-db, and your done!
* 822 formatted output for queues (http://ftp-master.debian.org/new.822); this information is now used on DDPO pages
* Rewriting DM management code to have more of a brain than the previous implementation.
What's next on the TODO list
* Content file generation from the database (part of removal of apt-ftparchive, but thats another blog post ;-)).
Oh, as a side note to my current readers, my blog has changed names to "Notes from Underground", after one of my favorite novels, and futher in reference to exploring the mysterious underground that is Debian's backend code. We're also now on Planet Debian :-).
Tuesday, November 18, 2008
ARM and MID ...
So for those of you playing along at home, we have a newly available ARM port, which has just reached the point of being able to debootstrap itself with a buildd varient (i.e., install build-essential and a few other packages). Now the fun begins with porting stuff.
First up is mono. Mono has a port to ARM and ARMel already, however, this code obviously hasn't been compiled in sometime, as it FTBFS due to glibc 2.8+svn. *grumble*. Oh well, easy fix, and one that will help clear a bunch of other dep-waits and get us that much closer to a usable port.
First up is mono. Mono has a port to ARM and ARMel already, however, this code obviously hasn't been compiled in sometime, as it FTBFS due to glibc 2.8+svn. *grumble*. Oh well, easy fix, and one that will help clear a bunch of other dep-waits and get us that much closer to a usable port.
Monday, November 10, 2008
WiiBuntu (Part 1), Ubuntu Ports MID, and lpia
So here I am, with a wonderful Wii, complete with Twilight Hack and Homebrew Menu installed, wondering if I could do something crazy like run Ubuntu on it. After some trial and error, it appears possible to do a debootstrap of an Ubuntu PowerPC installation and have it work!.
So now that I have a Wii with Ubuntu installed, what am I going to do with it, that is the question I am now pondering. The answer seems obvious, put Ubuntu MID on the sucker, install XWii (Wiimote Pointer Driver), and the WiiX display driver, and turn it into a very pretty multitouch system. Yay :-) So it should just be a matter of 'aptitude install ubuntu-mid'
Sadly, its not quite that easy. Ubuntu-MID currently is only installable on lpia. I can hear the questions now, WTF is lpia. To answer that question, lpia is the Low-Power Intel Architecture, also known as the Intel Atom processor family. For those of you familar with Atom, you might be double-taking, saying Atom is an x86 based processor. Again, your right, lpia is x86 based (you can run it on normal PCs), its essentially i386 Ubuntu with a few optimizations.
The problem breaks down to the way some of the MID's compontents were packaged. The main problem comes in the form that the rules file, instead of properly splitting the packages out, the rules file checks for the lpia architecture, and then changes the configure options to build hildon support (hildon is the nifty library from Nokia that makes MID work). To fix MID on ports (and x86/amd64), each package with this lpia detector switch must be found, and then properly split. Sounds easily enough, right?
Wrong. Most of the packages (such as evince) are CDBS packages. For anyone who knows anything about CDBS, you know it was never meant to split packages :-/. This *will* be fun :-/.
As an added note, OpenOffice.org is broken on PowerPC, double fun, since compiling it will likely take longer than releasing Jaunty will take :-).
So now that I have a Wii with Ubuntu installed, what am I going to do with it, that is the question I am now pondering. The answer seems obvious, put Ubuntu MID on the sucker, install XWii (Wiimote Pointer Driver), and the WiiX display driver, and turn it into a very pretty multitouch system. Yay :-) So it should just be a matter of 'aptitude install ubuntu-mid'
Sadly, its not quite that easy. Ubuntu-MID currently is only installable on lpia. I can hear the questions now, WTF is lpia. To answer that question, lpia is the Low-Power Intel Architecture, also known as the Intel Atom processor family. For those of you familar with Atom, you might be double-taking, saying Atom is an x86 based processor. Again, your right, lpia is x86 based (you can run it on normal PCs), its essentially i386 Ubuntu with a few optimizations.
The problem breaks down to the way some of the MID's compontents were packaged. The main problem comes in the form that the rules file, instead of properly splitting the packages out, the rules file checks for the lpia architecture, and then changes the configure options to build hildon support (hildon is the nifty library from Nokia that makes MID work). To fix MID on ports (and x86/amd64), each package with this lpia detector switch must be found, and then properly split. Sounds easily enough, right?
Wrong. Most of the packages (such as evince) are CDBS packages. For anyone who knows anything about CDBS, you know it was never meant to split packages :-/. This *will* be fun :-/.
As an added note, OpenOffice.org is broken on PowerPC, double fun, since compiling it will likely take longer than releasing Jaunty will take :-).
Wednesday, November 5, 2008
More notes from the PIE party (Ubuntu bootstrapping howto) ...
So after getting stuck and putting this work aside, I decided to take a second stab at it after meeting up with another Ubuntu developer in real life, and doing additional research. Using the experimental gcc-4.3 branch of Gentoo Hardened as a base, I'm now making extremely good progress bootstrapping amd64-pie, and the results look promising thus far.
For those of us curious, bootstrapping the archive is a straightforward if time intensive project. Essentially the process requires three individual bootstraps, an inital one that you use to build debian package, a chroot from those package, and a final chroot that is the end result.
Right now I'm working on the first part of this bootstrap, which is from a Linux host (without Debian) to generating the inital bootstrap packages. It requires compiling each build-dep from source with the proper configuration arguements, then building the packages with dpkg-buildpackage -d, and installing it, until you have build the entire base system.
From there, you take the debs, place them in a repo, and debootstrap, and then rebuild again, which produces the final result debs. It's straightforward and fasicating work (if a little tedious)
Michael
For those of us curious, bootstrapping the archive is a straightforward if time intensive project. Essentially the process requires three individual bootstraps, an inital one that you use to build debian package, a chroot from those package, and a final chroot that is the end result.
Right now I'm working on the first part of this bootstrap, which is from a Linux host (without Debian) to generating the inital bootstrap packages. It requires compiling each build-dep from source with the proper configuration arguements, then building the packages with dpkg-buildpackage -d, and installing it, until you have build the entire base system.
From there, you take the debs, place them in a repo, and debootstrap, and then rebuild again, which produces the final result debs. It's straightforward and fasicating work (if a little tedious)
Michael
Subscribe to:
Posts (Atom)
Posts
