LUG talks today

Today I gave three talks at my local LUG. The first was my latest SE Linux talk (I’ll put the notes online soon). The second was a talk about voting.

I asked for a show of hands, who has already decided which party they will vote for at the next federal election (about 12 people put their hands up). I then asked people to put their hands down if they were not a member of the party that they intend to vote for, including myself there were only two raised hands in the room (including mine)!

...

The final talk I gave was about getting speakers for Linux Users’ Groups.

The full post is here.

Linux Tour Bus

I have seen buses used for tours that contain bunk beds. If one or more such buses were hired then a group of Linux people could go on a moving Linux conference. This would have to take place in an area with many reasonable size cities in a close area and where there is a good number of Linux people in such cities. Probably the EU is the only area.

A bus (or several buses depending on demand) would then take a group of Linux people through the major cities and have a conference in each one.

Currently there are conferences such as the Debian conference DebConf which receive sufficient sponsorship money to pay for many speakers to attend. Having a similar conference traveling around Europe shouldn't cost any more money and will give plenty of time for the people in the bus to do some coding along the way.

We already have the Geek Cruises, my idea is to do a similar thing but on the road. Also it isn't practical to transport an entire conference, so it would probably just be speakers on buses and the audiences would vary from city to city.

BLUG

This weekend I went to the Ballarat install-fest, mini-conf, and inaugural meeting of the Ballarat Linux Users' Group (BLUG).

This was the second install-fest, the first one was quite successful so it was decided that there was demand for a second. I suggested that what we should do is get some of the more experience members of LUV to attend and give talks about their areas of expertise and make a mini-conference. I also suggested that we
hire a large vehicle to take a number of people to the meeting. Both my suggestions were accepted.

So on Friday evening I was in a Kia XXX with five other people from LUV on our way to Ballarat.

On Saturday we had the install-fest. We started at about 10AM, there were about a dozen people getting help installing Linux and many more attending the mini-conf and just hanging out. For lunch we had a BBQ. In the afternoon I gave a talk on SE Linux and then a brief impromptu talk on Poly-Instantiated Directories while the next speaker was setting up their laptop.

At the end there was the inaugural meeting of BLUG. The president was appointed, and there were some brief discussions about when to schedule meetings. I suggested that BLUG meetings should be either the day before or the day after LUV meetings to increase the incidence of speakers from other regions attending both meetings, my suggestion was being seriously considered at the time the meeting adjourned - LUV is a larger group and has better ability to get speakers from other regions. It was also agreed that a
weekend combined LUV and BLUG meeting would be arranged twice a year.

I traveled back to Melbourne by train which was cheap at $9 and comfortable. There was even a power point in the carriage (which I didn't use as my laptop was charged and the location was not convenient). For the next such event I'll try and arrange a group to travel on the train together.

The next thing to do is to find other regional centers in Victoria where we can do the same thing. Bendigo might be a possibility.

Also if you are a member of a LUG in a city please consider the possibilities for helping form a LUG in a regional center that's nearby. I would be happy to provide whatever advice I can to help people replicate this success in areas surrounging other cities, so please email me if you have any questions.

meeting people at Linux conferences

One thing that has always surprised me is how few people talk to speakers after they have finished their lecture. A lecture might have many questions and the questions may be cut off, but when the speaker leaves the room they will usually do so alone.

When I give lectures at conferences I'm always happy to spend more time talking to people who are interested in the topic and disappointed that so few people choose to do so. It seems that other people have similar experiences, there have been several occasions when I have invited speakers to join me for lunch and no-one else has shown interest in joining us.

Usually the most significant factor in making someone offer a talk at a Linux conference is the opportunity to teach other people about the technology that they are working on. People with that motivation will take the opportunity to teach people at lunch, dinner, whenever.

Linux Conf Au has an event called the "Professional Delegates Networking Session" which is regarded by some people as the way to meet speakers (about half the delegates don't attend so the ratio of speakers to delegates is significantly better than at other conference events). But it seems to me that it's more efficient to just offer to buy them dinner. When I worked for Red Hat the maximum value for a gift I could accept was $100US, I expect that Red Hat has not changed this policy since then and that most companies that employ speakers at Linux conferences have similar policies. $100US is more than a meal costs at most restaurants that are near a Linux conference.

If I was a manager at a company that sent employees to a Linux conference I would first send email to some speakers who were working in areas of Linux development that were related to the projects that the employees were working on. I would ask the speakers if they would be interested in having dinner bought for them by my company and give them the option of bringing one or two friends along for a free meal (the friends would probably be people who work in similar areas).

some random Linux tips

• echo 1 > /proc/sys/vm/block_dump
  
  The above command sets a sysctl to cause the kernel to log all disk writes. Below is a sample of the output from it. Beware that there is a lot of data.
  
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX152 on dm-6
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX160 on dm-6
  Jan 10 09:05:53 aeon kernel: kjournald(1048): WRITE block XXX168 on dm-6
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX384 on dm-7
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX400 on dm-7
  Jan 10 09:05:54 aeon kernel: kpowersave(5671): READ block XXX408 on dm-7
  Jan 10 09:05:54 aeon kernel: bash(5803): dirtied inode XXXXXX1943 (block_dump) on proc
  


• Prefixing a bash command with ' ' will prevent a ! operator from running it. For example if you had just entered the command " ls -al /" then "!l" would not repeat it but would instead match the preceeding command that started with a 'l'. On SLES-10 a preceeding space also makes the command not appear in
  the history while on Debian/etch it does (both run Bash 3.1).
  


• LD_PRELOAD=/lib/libmemusage.so ls > /dev/null
  
  The above LD_PRELOAD will cause a dump to stderr of data about all memory allocations performed by the program in question. Below is a sample of the output.
  
  

  Memory usage summary: heap total: 28543, heap peak: 20135, stack peak: 9844
          total calls   total memory   failed calls
  malloc|         85          28543              0
  realloc|         11              0              0   (in place: 11, dec: 11)
  calloc|          0              0              0
    free|         21          12107
  Histogram for block sizes:
     0-15             29  30% ==================================================
    16-31              5   5% ========
    32-47             10  10% =================
    48-63             14  14% ========================
    64-79              4   4% ======
    80-95              1   1% =
    96-111            20  20% ==================================
   112-127             2   2% ===
   208-223             1   1% =
   352-367             4   4% ======
   384-399             1   1% =
   480-495             1   1% =
  1536-1551            1   1% =
  4096-4111            1   1% =
  4112-4127            1   1% =
  12800-12815           1   1% =
  

more about Fedora

In a comment on a previous blog entry I was described as an active Fedora advocate, I don't think that is an accurate description. I advocate it to appropriate people, which is mostly non-programmers - but as I mentioned that means a larger proportion of the population than to whom I can advocate Debian. It's not that I'm trying to advocate Fedora, just that it fills a need for many people. I believe that the term Fedora advocate means someone has an objective of increasing the use and to use Fedora, I don't have such an objective. I am a Linux advocate, a Free Software advocate, and sometimes a Unix advocate (Unix meaning the entire family of Unix-like operating systems). Merely promoting something does not make you an advocate for it. I don't think of myself as a Debian advocate at this time, but as I am a Debian developer this may change.

It seems that the people who run the Fedora Planet think that my blog has suitable Fedora content, it's been added to that planet. Also the Fedora Planet appears to be running an older version of the Planet software as it has the same problem with my blog that Debian Planet had before the upgrade.

Now on the issue of gratis vs libre: As I am not a Red Hat employee I can't maintain a kernel-xen-nopae package and give it the same status as the kernel-xen package. Even when I was a Red Hat employee I couldn't have done that - it would require some amount of management approval. I believe that this fundamentally makes Fedora less of a libre distribution. There is no room in Fedora for someone who is an upstream developer and who just wants to maintain their own package. There is Fedora-Extras, but that has a second-class status. Only Red Hat employees can maintain packages in Fedora Core. This makes Fedora fundamentally less libre than Debian. I am not trying to suggest that Red Hat change things in this regard, I believe that Fedora is meeting all it's goals and that making Fedora as libre as Debian is not possible given the goals of making a profit on selling support of RHEL.

Chris made a good point. I also believe that MP3 codecs should not be in Debian/main. But I believe that people making mistakes about some issues is not a factor in judging the entire project. I believe that Debian is more libre although some bad decisions were made - largely due to lack of overall management. Fedora has hierarchical management, so when the legal team declares that some software can not be distributed then it gets removed without debate. I guess I could propose a GR to exclude MP3 codecs from main.

Also it should be noted that RHEL Extras has some of this software that is not in Fedora (RealPlayer for example). The Red Hat legal advice was that MP3 codecs need a license, so they ship a licensed version in their commercial distribution. This is the right thing to do for their customers (it's handy to have and I'm sure that they get a good deal by paying license fees for all their customers) and removing such things from Fedora is the right way to offer a gratis product without unreasonable legal liability.

Naturally Fedora is much more libre than any secret-source OS. Every user has the option of downloading the Fedora source and recompiling it as they wish. I could compile Fedora with a Xen kernel that runs on my hardware and with SE Linux policy that is more restrictive than that which Fedora currently has. I could build custom Fedora install CDs to install things the way I want (which I considered doing when I worked for Red Hat). But the liberty to fork a project does not compare to the liberty to join it, and the liberty to create your own packages in extras does not compare to the liberty to add your own packages that do things differently to the default package.

There are of course positive and negative aspects to this. I started work on SE Linux in Debian in 2001. In 2003 I joined Red Hat to work on SE Linux, in Red Hat I was not the only person dedicated to SE Linux work and other people spent part of their time working on it. The SE Linux work in Red Hat soon eclipsed that of Debian because there was management support. There was no possibility for a package maintainer to refuse to fix a bug that affected SE Linux simply because they didn't care for it. The positive side of this is that the SE Linux work proceeded quickly and efficiently. The negative side of this is that things which don't have management support don't appear in Fedora Core. Exim is a fine MTA but is not in Fedora Core. Some people think that AppArmor is a better option than SE Linux, they are wrong - but in Debian any developer has the option to add AppArmor support and neither I nor any other DD can prevent them. The libre nature of Debian means that as long as basic technical criteria are met DDs can add any package that they wish to the distribution.

These issues however are all related to people who are actively involved in Free Software development. For a typical Free Software user it often doesn't make much difference, until of course your favourite program doesn't get management approval to appear in Fedora Core. But the counter argument is that the quality of some of the >10,000 packages in Debian is not so high. You can install a Fedora Core package and have a reasonable expectation about how well it works, but Debian packages are sometimes rather experimental.

I also don't believe that Debian is a very functional Democracy. Some of the problems of Direct Democracy are demonstrated in Debian. In many ways it is more anarchistic, anarchy gives you liberty for good and bad. Maybe we should consider a Representative Democracy model for Debian.

The benefits of SE Linux

Today I discovered a bug in one of my programs, it called system() and didn't correctly escape shell eta-characters. Fortunately I had written custom SE Linux policy for it which did domain_auto_trans(foo_t, shell_exec_t, very_restricted_t) so there was no possibility of damage.

The log files (which were not writable by the daemon by both SE Linux access control and Unix permissions) indicated that no-one had attempted to exploit the bug.

dunc-tank and motivation

The dunc-tank project was established to raise money to compensate some Debian developers who are essential to producing a timely release of Debian. There has been a lot of acrimoneous debate about whether this is a good or bad thing. The positive side of it is that the release managers will get to spend more time working on Debian, the negative side is that some volunteers will lose motivation.

However I have felt more motivated to do my unpaid Debian work. During the time that I was employed by Red Hat I was fairly slack about my Debian development work (incidentally Red Hat management were happy for me to continue Debian work so there was no pressure from Red Hat in this regard). Since leaving Red Hat I have been busy doing paid work.

Recently I have started getting involved in Debian work again. I am about to upload a new version of Postal for the first time in three years, I have set up a Xen server for Debian SE Linux development, and I am about to start serious Debian SE Linux development work again.

One factor in this has been my impression that other DDs are taking the release seriously. In the past schedules for release have slipped repeatedly without end. Now there is a schedule and this gives me more motivation to get bugs fixed!

Ruxcon and SLUG

This weekend I was in Sydney for Ruxcon. Ruxcon is a computer security conference with a focus on penetration testing and related skills.

The presentation on Unusual Bugs by Ilya van Sprudel was particularly interesting. He spoke about a number of issues that could do with some improvement in Linux, I will file some bug reports shortly.

There was a chilli eating contest. I was one of six people to enter. I survived the first two rounds and got onto the middle-strength chilli before giving up. There were 100 tickets to the Google party for the ~200 person conference and everyone who entered a contest got a ticket. My aim in the contest was to eat more chilli than I enjoy eating but less than the amount required to make me sick, with a secondary goal of tasting at least the second level of chilli. I achieved my goals and left the contest after tasting the second chilli.

One man appeared to be impressed by my chilli eating and was telling everyone that I am famous for eating chilli. It's good to be famous for something in the computer security community. :-#

At the end of the conference there was a panel discussion that I was invited to attend. I had to leave early to catch my flight, at the time I left everyone who was on the panel had each finished a few drinks and a couple of new guys had just joined. I think I missed the most exciting part of the panel discussion.

Thanks to whoever paid for the drinks for panel members. Things were a little hectic when we were given the drinks and I forgot to thank whoever paid for them.

In other news Sydney trains are slow and unreasonably expensive, $13 to get from the airport to the SLUG meeting at St. Leonards seems excessive. With all the problems with Sydney roads they really need to get a better public transport system!

While in Sydney I attended a SLUG meeting and gave a short talk about Postal (my mail server benchmark suite). I will present a paper about Postal at the OSDC conference later this year.

SAK, ctrl-alt-del, and Linux keyboard mapping

A common problem with Linux systems is when Windows users press CTRL-ALT-DEL at the login prompt and reboot the machine.

To fix this some people change the ^ca line in /etc/inittab to just disable the reboot function. However this is not desirable because sometimes you want to reboot a machine with a simple keypress.

Another problem that has not been widely considered is the use of fake login prompts by attackers. This can be implemented in either text mode or graphics mode. All the fake login prompt has to do is display something that looks like a real login prompt, accept a user-name and password, verify the password (a localhost ssh connection is a good way of doing this) and then abort. In the case of a text-mode login the user will think that they entered the wrong password, in the case of a GUI login via an XDM program the user will think that the login program just crashed. Then the attacker has access to their account.

The solution to the fake-login problem is the use of the Secure Attention Keyboard (SAK) feature. When invoked this feature makes the kernel kill all processes that are on the virtual console in question. If you make CTRL-ALT-DEL the SAK combination then pressing those keys will cause the kernel to kill any processes that are attached to the current virtual console and preventing the ability of hostile programs to forge a login prompt (which is the same as it's purpose in Windows).

The next thing to do is to make another combination used for system boot. A reasonable combination seems to be CTRL-ALT-BREAK as those keys are widely separated and the combination is not used for anything else.

If you put the following in a file named sak.map (or whatever you want to call it) then the command loadkeys sak.map will apply the change. Note that when creating a keyboard map you should do it on a machine for which you don't mind being forced to perform a hardware reboot. It's easy to make a mistake and give yourself a keyboard mapping that is not usable. Another possibility is to do such testing on a machine that allows ssh logins, you can then login via ssh and run loadkeys -d to correct any errors you might make.

control alt keycode 119 = Boot
control alt keycode 83 = SAK
control alt keycode 111 = SAK
control altgr keycode 119 = Boot
control altgr keycode 83 = SAK
control altgr keycode 111 = SAK

Note that the above covers both ALT and ALT-Gr keys as well as the numeric keypad and regular versions of the delete key.

dumpkeys -l gives you a list of all possible keyboard combinations. showkey will display the number matching any key you press and will exit after 10 seconds of inactivity.

tcpdump and ps

Today I was doing some network tracing and figured out how to track the start and end of TCP connections. The following tcpdump command will get all SYN, FIN, and RST packets on port 80 and all ICMP packets:

tcpdump -i bond0 -n "port 80 and tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0 or icmp"

Also recently I was tracking down some minor security issues related to programs that call setuid() to drop privs but never call setgid() and therefore always run with GID==0 which gives them a lot of access to the system. The following ps command gives the real, effective, saved, and filesystem UIDs and GIDs mapped to names. Note that with some versions of ps different fields have different truncation lengths.

ps -eo pid,user,euser,suser,fuser,group,egroup,sgroup,fgroup,comm

The next thing I have to do is to patch PS to show the supplementary groups.

Ethernet bonding

Bonding is one of the terms used to describe multiple Ethernet cables used to form a single virtual network link. This can be done for performance or reliability.

Bonding for performance used to be common when 100baseT was the fastest network technology that was commonly available. In 1999 servers could usually sustain considerably more than 10MB/s so a single 100baseT network interface was a performance bottleneck. At that time I worked with Cisco switches and Solaris machines that had up to four 100baseT links bonded for performance.

Nowadays Gigabit Ethernet is commonly available, most laptops have Gigabit Ethernet on the motherboard. Gigabit PCI cards are as cheap as $35, and Gigabit switches can be purchased for as little as $139. Server hardware is a little more expensive, but it's still quite cheap and commonly available.

Most people don't need more than Gigabit speed, in fact most systems can not saturate a Gigabit link due to poor application design, a slow operating system, or slow disks used to provide the data. So at this time there is little speed for bonded Gigabit networking for performance.

There is still the issue of reliability. Often you want to have two ethernet cards and cables configured so that if one breaks the network won't go down.

One annoying thing about bonding in Linux (in 2.6.x kernels) is that the module has to be loaded separately for each bond interface, and the parameters for an interface can't be changed without unloading and loading the driver (very painful if you log in to the machine via ssh over the bonded interface to do sys-admin work).

The parameters I have in /etc/modprobe.conf for bonding are:

alias bond0 bonding
options bond0 mode=1 arp_interval=500 arp_ip_target=192.168.0.1

This means that if there is no traffic on the link then every 500ms an ARP request will be sent for the address 127.128.129.130 (I used the address of my router but substituted a different value for this blog entry). An ARP request for a machine on the local LAN is a request that will always be satisfied if the machine in question and the network link are working.

The idea is that you have two switches and every computer that matters has two ethernet ports. If one port stops working (broken Ethernet card, cable, or router) then the other takes over.

The special file /proc/net/bonding/bond0 can be used to view the current configuration of the bond0 device.

Below are sample configuration files for Fedora and Red Hat Enterprise Linux to configure bonding:

/etc/sysconfig/networking/devices/ifcfg-bond0:
DEVICE=bond0
IPADDR=192.168.0.2
NETMASK=255.255.255.0
NETWORK=192.168.0.0
BROADCAST=192.168.0.255
ONBOOT=yes
BOOTPROTO=static
# GATEWAY should be the IP address to ARP ping
GATEWAY=192.168.0.1
TYPE=Ethernet

/etc/sysconfig/networking/devices/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
SLAVE=yes
MASTER=bond0
TYPE=Ethernet

/etc/sysconfig/networking/devices/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
HWADDR=xx:xx:xx:xx:xx:xx
ONBOOT=yes
SLAVE=yes
MASTER=bond0
TYPE=Ethernet

Note that there is nothing preventing you from having more than two devices bonded together for reliability, but I doubt that you really need that.