Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 22 Jul 2013 01:00:04 -0500
From: Rob Landley <>
Subject: Re: embedded newbies site.

On 07/16/2013 06:50:29 AM, LM wrote:
> On Mon, Jul 15, 2013 at 10:03 PM, Rob Landley <> wrote:
> > I'd like an explicit a place to collect and preserve information  
> about
> > this sort of thing, and a place we can send newbies to ask all the  
> stupid
> > questions. The main page should teach somebody what embedded  
> development
> > _is_ and how to do it, starting with how to build and install the  
> simplest
> > Linux system that boots to a shell prompt (three packages: linux,  
> musl, and
> > toybox).
> >
> Sounds like a great idea.  Would be interested in reading articles on  
> some
> of the topics mentioned.  Sites like state what they  
> consider
> are better and worse software choices.

I lurked on #suckless irq channel on OTP or whatever it was for a week.  
It seems to be support for some window manager (dwm?). Nothing else was  
ever discussed...

> Would be nice to see some actual
> statistics and rationale backing up what is considered better or worse
> design.  For instance, there are some negative mentions about the PCRE
> library, but when I tried to track down the cons for using it, I only  
> found
> dated performance comparisons showing how poorly it worked if you  
> don't use
> the newer JIT implementation.

The great thing about Linux From Scratch is it's practical. It's a  
procedure you can actually reproduce for yourself, and when you try it  
you get a running system that you built and are in a position to  
modify. It mostly explains why you're doing what you're doing, and  
provides some alternatives along the way.

But Linux From Scratch 3.x was a better learning experince than the  
current one, because these days it's much biger and much more  
complicated to get a running system, and you don't really learn much  
more. Plus the "hints" files about things like BSD init scripts are  
sort of deprecated now. And it doesn't really present stuff like tcl as  
optional, even though it's only ever used to run test suites...

Beyond Linux From Scratch is about adding stuff to the base linux  
system, but there's nothing in there about _removing_ stuff. Or  
swapping out base packages for alternatives. (Again, the "hints" used  
to go into this, but they seem to have tailed off over the past few  

Oh, we should totally be linking to and  
possibly trying to reproduce a current version under 3.x kernels.

A lot of stuff, anybody can take and just do the legwork. For example,  
we really need a current version of and somebody could  
just _take_ that and compare it with the current kernel and do an  
updated version based on what they learn by reading current kernel  
source using the old 2.4 version as a guide...

> What might be a positive for a system that's
> optimized for a particular processor might be a negative if you're
> interested in software that ports to multiple processors and vice  
> versa.

I've yet to find a per-processor optimization that buys you one  
interation of Moore's Law.

And I _have_ seen years of seesawing back and forth over "here's a  
lookup table of 256 angles your ship can be at where we've done all the  
triginometry... oh the new processor has a floating point coprocessor  
and tiny l1 cache so it's actually faster to calculate it than thrash  
the cache with our lookup table, oh now the NEW processor has a big L2  
cache so the lookup table is faster again, oh now they've added 3D  
hardware so all this mess is slower than having the 3D hardware do  
it..." I've seen optimizations where the pendulum went back and forth a  
half dozen times on the same issue with RISC vs CISC and whether loop  
unrolling is a win and...

And what it keeps coming back to is "simple code you understand beats  
clever code you don't". Do a simple implementation, let the compiler  
optimize it for you, the bit you're doing is not the hard part (if it  
is, you're doing it wrong, which means back up and rethink), so just  
get it done and stay out of the way while Big Important Programmers do  
things they find Terribly Hard To Do because they're straining at gnats  
and all that...

(Premature optimization is the root of all evil, when in doubt use  
brute force, etc.)

> Musl's useful not just for embedded systems but for older machines  
> that
> want to run efficient desktop environments.  However, what works for a
> desktop environment might not work well for an embedded system and so  
> on.

Knoppix was a fine desktop. I used it as such for over two years. I  
installed it to the hard drive because running from CD was slow and  
tying up the CD drive was inconvenient, but operating under a space  
constraint on the image size meant they had to figure out what they  
really NEEDED, and it made a better system.

> Would like to see actual lists of pros and cons, less opinions and  
> let the
> user decide if the software is really a bad fit with his/her needs or  
> not.

I wouldn't presume to advise people without knowing what they wanted to  
use a system _for_. For an awful lot of people Red Hat Enterprise is  
the correct choice, albeit for financial rather than technical reasons.  
(You know why Red Hat drove Sun Microsystems out of business, right?)

> Would also love to see a forum where one could discuss pros and cons  
> of
> various software and library choices, alternatives already out there  
> and if
> the user wants to rewrite some of these himself or herself for  
> specific
> needs, a place to discuss design issues.

I'm not sure you're asking well-defined questions here.

Ok, simple mindset: complexity is a cost. Treat complexity as something  
you spend to get functionality, and you're 3/4 of the way to making  
good decisions.

There's some fuzziness in measuring complexity, but lines of source  
code maps pretty well to "amount of human thought required to  
understand what this thing is doing". If you have a project that gets  
the job done with 100,000 lines of code, and another one that requires  
2 million lines of code, to _me_ it's pretty darn clear which of the  
two is superior.

You can then say 'but the big one has features X, Y, and Z I need, and  
we benchmarked the performance in our deployment environment and the  
big one performs 12.7% faster", and then you go "do you really need  
those features? How much work would adding them to the small one be,  
and would the upstream project take it or just consider it bloat, and  
if you were to maintain your own patchset to add that feature to the  
small one would it change your answer about whether or not you actually  
need it?"

And of course there's complexity you directly engage with and  
complexity you indirectly engage with; your _local_ complexity may be  
"this giant black box works for everybody else so just using it is very  
easy for us as long as it never breaks, and if it does we have a vendor  
for that". And of course if you _are_ the vendor, deploying dropbear  
instead of openssh can have a negative PR effect because openssh is The  
Standard but if that's such a big deal why aren't you using Windows...

And really all this infrastructure is generally stuff that should just  
work, and should be an existing solved problem so you can focus on your  

See how it spirals through a gazillion topics? As I said: not sure what  
questions you're really asking.

> There is an lfs-chat list.  Think it would probably be a good idea to  
> post
> something about the idea of an LFS for embedded systems there and see  
> if
> any of the regular LFS users would be interested in getting involved.  
> A
> start might be to take the outline of possible topics Rob Landley  
> supplied,
> put it up on a wiki and see if people will volunteer to fill in some  
> of the
> blanks.  Might also be useful to get together a list of what tasks  
> need to
> be done to get something started and ask for actual volunteers for  
> each
> task to help get things rolling.  I do think a mailing list or forum  
> would
> be useful as well.  That way, one can get discussions going and  
> brainstorm
> ideas about how best to program something or find information on a  
> topic.
> I tend to prefer mailing lists and forums to IRC.  It's easier to read
> through past information.

Good concrete questions to answer are a good start. Not "maybe people  
would want to know X" but "I want to know X."

> I've been talking with another developer about the possibility of  
> building
> (yet another) lightweight Linux distribution for older machines.  I  
> really
> haven't been happy with what's currently out there.

Aboriginal Linux is the lightest-weight development environment I know  
how to make. (And switching to musl should make it lighter.)

> The average definition
> of a lightweight Linux desktop for older machines is to use a lot of  
> GTK+
> programs (with a lightweight desktop like XFCE (not my definition of
> lightweight), LXDE or razorQT) and even interpreted programs (as long  
> as
> they look like they're in console mode or like they might somehow be
> lighter or more useful than their compiled equivalents).

X11 is a windowing system. It draws graphics onna screen; lines, fonts,  
boxes, stamping images from bitmaps and such, and the bitblts and  
double buffering used for dragging and scrolling windows and such.

Then you have a window manager that draws borders and title bars and  
menus, and gives them behavior so when you grab the corner and drag it  
the window resizes, or grab the title bar the window moves, or handles  
the z-order stuff so windows draw in front of other windows (which  
pragmatically means you hide or clip window areas and only draw parts  
of 'em).

Then you have a toolkit, which is a shared library of graphics  
primitives and associated behavior when they get mouseovers or clicks  
or keys on the keyboard are pressed while it has focus. (Window manager  
defines what "focus" is and sending keypresses and clicks to the right  
thing.) Your toolkit is where you find code to implement a button or a  
scrollbar or a pulldown menu.

Then you have a desktop program, which is the thing that runs to _use_  
X11, a window manager, and a toolkit to provide behavior for an  
otherwise empty screen. It provides the bar alogn the top that shows  
you your list of pen windows, and provides a menu of stuff you can  
launch, and a dock for tiny icons associated with programs that know  
about that type of desktop and can do the right transaction with it to  
register themselves.

I'm running the xubuntu linux distro. It's using xfce as the desktop  
program, which uses the gtk toolkit, and xfwm4 is the window manager.  
All running on top of which is the windowing system.

It's possible _not_ to use all these layers of stuff, but generally  
when a program doesn't it's because its reinventing it. You don't have  
to use gtk, you can have your program draw its own buttons and respond  
to mouse clicks in that area of its window manually: and that means no  
two programs look or behave remotely the same.

Once again, defining "simple" requires understand what it is you're  
trying to _do_. Simple is an adjective, not a noun.

> They typically
> use the KISS principle which means (according to their take on it) I'm
> stuck with the one graphics editor, the one music player, etc. that  
> the
> distribution creator happens to like.  A Gimp or a Photoshop style  
> program
> has a lot of functionality.  So does an Office Suite like LibreOffice.

Star Office (a german company that Sun bought and renamed Open Office)  
was the first non-microsoft program that actually had good support for  
reading and writing Word files, due to YEARS of careful reverse  
engineering effort that started back on OS/2 before being ported to  

The opening of Open Office had the same failure mode Mozilla did (long  
story, I did a talk about it once if you're bored) and the resulting  
code bloat is epic. But getting the "reads, edits, and writes word  
documents well" functionality out of anything _else_ turns out to be  
really hard.

> If you're going to replace heavyweights with a program that does one  
> thing
> well, you're typically going to need more than one application with  
> each
> application designed to perform a specific piece of the functionality
> well.  You need more than one type of graphics program if you're doing
> serious graphics editing, more than one type of music program if  
> you're
> doing serious music creation, etc.  A lot of the topics such as how  
> to put
> together a system from scratch, what boot and init programs to go  
> with,
> which userspace utilities to use, which package manager to use, which
> libraries are efficient would be of great interest for the project.

Linux From Scratch and Beyond LInux From Scratch already cover this.  
And Gentoo set about trying to automate it. Both have serious failings,  
but they're an existing starting point to acquire this knowledge.

What neither does is says how to set up a simple base system that isn't  
infested with gnu crap, and then extend it towards providing the  
prerequisites packages such as OpenOffice require. Learning how to swap  
busybox for coreutils and make that work to run postgresql on the  
resulting system...

> Another concern to me is which projects are open to accepting patches  
> and
> which aren't so open, making it prudent to look into more friendly
> alternatives.  I'd also been interested in discussing when it pays to
> rewrite something from scratch and when it's better to reuse what's  
> already
> been done.  I've been picking up ideas by looking at the code embedded
> systems use.  However, the end goal for this particular project is  
> not an
> embedded system but a GUI desktop that an average end user will be
> comfortable working with.  There's a lot of overlap, but definitely
> different goals with different design tradeoffs.

Embedded and non-embedded systems are distinguishable by the  
"complexity is a cost" mindset. Desktop systems seem to think they have  
unlimited storage, memory, bandwidth, processing power, and so on due  
to Moore's Law, and that they also have unlimited warm bodies capable  
of maintaining the result due to open source and the internet.

Embedded systems are designed with the idea that fitting those 15  
million lines of code into 30 cents worth of flash memory could be  
painful. That running it on a processor running off a watch battery may  
be slow. That one junior engineer alotted 3 days to port it and every  
prerequisite package it requires to a brand new processor implemented  
in an FPGA with a beta compiler fork based on gcc 3.4 might have a  
rough time of it. That there's some exploit lurking in those 15 million  
lines of code, and when you put it on a system that no human being will  
log into for two years, that doesn't get upgraded in all that time, but  
has a broadband connection to the internet, bad things will happen.

Think about Mozilla vs webkit. Mozilla is based around the idea that  
writing a good browser is hard, and there should be only one, and it  
must do everything for everybody and be perfect.

Webkit is based on the idea that a browser is disposable and gets  
reimplemented from scratch every few years. Webkit started life as the  
KHTML engine in Konqueror, the browser built into KDE which went from  
zero to usable in about a year. Then Apple grabbed it and forked it and  
did Safari out of it. Then Google grabbed it and forked it and did  
Chrome out of it. I expect in a couple years people will throw chrome  
out and do a new one.

Google designed Chrome to work with Android, on phones and tablets. You  
can kill individual tab processes, because they acknowledge they're  
going to break and it won't be perfect so that's a thing you may want  
to _do_. It's got a lot more of the embedded mindset than Mozilla, as  
Scott McCloud explained back at the launch:


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.