Showing posts with label Unix. Show all posts
Showing posts with label Unix. Show all posts

Friday, January 6, 2017

Few thoughts about systemd and human behavior

I was just reading comments on a post on Hackernews about systemd. Systemd, as you might know, is a replacement for the venerable init system. Anyway, reading the comments was reading about all the same story over and over again. Namely, there are those strongly pro and those strongly con the systemd, in some cases based on arguments (valid or not) and in other cases based on feelings. In this post I won't go into technical details about systemd but I'll concentrate on a human behavior that is the most interesting to me. And yes, if you think I'm pro systemd, then you're right, I am!

Now, what I think is the key characteristic of people is that they become too religious about something and thus unable to critically evaluate that particular thing. It happened a lot of times, and in some cases the transition from controversy was short, in other cases it took several or more generations of human lives. Take as an example the Christian religion! It also started as something controversial, but ended as a dogma that isn't allowed to be questioned. Or something more technical, ISO/OSI 7 layer model. It started as a controversy - home many layers, 5, 6, or 7? The result of this controversy we know, and after some short period of time it turned into a dogma, i.e. that 7 layers is some magical number of layers that isn't to be questioned. Luckily, I don't think that it is the case any more, that is, it is clear that 7 layers was too much. Anyway, I could list such cases on and on, almost ad infinitum. Note that I don't claim that any controversial change succeeded in the end, some were abandoned and that's (probably) OK.

I should also mention one other interesting thing called customs (as in norm). People lives are intervened with customs. Anyway, we have a tendency to do something that our elders did just because, i.e. we don't know why. I don't think that's bad per se, after all, probably that helped us to survive. The problem with the customs is that they assume slow development and change in environment. In such cases they are very valuable tool to collect and pass experience from generation to generation. But, when development/change speed reaches some tipping point, customs become a problem, not an advantage - and they stall adjustment to new circumstances. So, my personal opinion about customs is that we should respect them, but never forget to analyze if they are applicable/useful or not in a certain situation.

Finally, there is one more characteristic of a human beings, and that is inertia. We used to do things in certain way, and that's hard to change. Actually, I do not think that it is unrelated to religion and customs, actually on the contrary, I think they are related and it might be something else that is behind. But i won't go into that, at least not in this post.

So, what all this has to do with the systemd? Well, there is principle or philosophy in Unix development that states that whatever you program/create in Unix, let it do one thing and let it do it right. For example, tool to search for file should do it well, but not do anything else. And my opinion is that this philosophy turned into a custom and a religion in the same time. Just go through the comments related to SystemD and read them a bit. A substantial number of arguments is based on the premise that there is a principle and it should be obeyed under any cost/circumstance. But all those who bring this argument forget to justify why this principle would be applicable in this particular scenario.

And the state of computing has drastically changed from the time when this philosophy was (supposedly) defined (i.e. 1970-ties) and today's world. Let me say just a few big differences. Machines in the time when Unix was created were multiuser and stationary, with limited resources and capabilities, and they were used for much narrower application domains than today. Today, machines are powerful and inexpensive, used primarily by a single user. They do a lot more than they used to do 40 years ago, and they offer to users a lot more. Finally, users expectations from them are much higher than they used to be.

One advantage of doing one thing and doing it well was that it reduces complexity. In a world when programming was done in C or assembler, this was very important. But it also has a drawback, and that it is that you lose ability to see above the simple things. This in turn, costs you performance but also functionality, i.e. what you can do. Take for example pipes in Unix. They are great for data stored in text organized in a records consisting of lines. But what about JSON, XML and other complex structures? In other word, being simple means you can do just a simple things.

This issue of simple and manageable and complex and more able is actually a theme that occurs in different areas, too. For example, in networking where you have layers, but because cross layer communication is restricted means you have a problems with modern networks. Or, take for example programming and organizing software in simple modules/objects. Again, the more primitive base system is, the more problems you have to achieve complex behavior - in terms of performance, complexity, and so on.

Few more things to add to the mix about Unix development. First, Unix is definitely success. But it doesn't mean that everything that Unix did is a success. There are things that are good, bad, and ugly. Nothing is perfect, nor will ever be. So, we have to keep in mind that Unix can be better. The next thing we have to keep in mind is that each one of us has a particular view on the world, a.k.a. Unix, and our view is not necessarily the right view, or the view of the majority. This fact should influence the way we express ourselves in comments. So, do not over generalize each single personal use case. Yet, there are people who's opinion is more relevant, and that are those that maintain init/systemd and similar systems, as well as those that write scripts/modules for them.

Anyway, I'll stop here. My general opinion is that we are in 21st century and we have to re-evaluate everything we think should be done (customs) and in due course not be religious about certain things.

P.S. Systemd is not a single process/binary but a set of them so it's not monolithic. Yet, some argue its a monolithic! What is a definition of "monolithic"? With that line of reasoning GNU's coreutils is a monolithic software and thus not according to the Unix philosophy!

Friday, January 9, 2015

Getting free disk space in Linux

While working on a script to have full Zimbra backups as many days in the past as possible, I was trying to automatically remove old backups based on the free space value. Basically, the idea was to remove directory by directory until free space reached some threshold. To find out free space on a disk is easy, use df(1) command. Basically, it looks like this:
$ df -k /
Filesystem 1K-blocks     Used Available Use% Mounted on
/dev/sda1   56267084 39311864  16938836  70% /
The problem is that it is necessary to use some postprocessing in order to obtain desired value, i.e. 5th or 5th column. cut(1) command, in this case, is a bit problematic because in general you can not expect that the output is so nicely formatted, nor it is fixed. For example, based on the width of the widest device node in the first column, it is automatically resized. That in turn means number of whitespaces varies, and you end up being forced to use something else than cut(1). Probably, the most appropriate tool is awk(1), since awk(1) can properly parse fields separated with variable number of whitespaces. In addition, you need to get rid of first line. That can be done using head(1)/tail(1), but it is more efficient to use awk(1) itself. So, you end up with the following construct:
$ df -k / | awk 'NR==2 {print $4}'
16938836
But, for some reason, I wasn't satisfied with the given solution because I thought I'm using too complex tools for something that should be simpler than that. So, I started to search is there some other way to obtain free space of some partition. It turned out that stat(1) command is able to do that, but it's rarely used for that purpose. It is used to find out data about files, or directories, but not file systems. Yet, there is an option, -f, that tells stat(1) we are querying file system, and also there is an option --format which accepts format sequences in a style of date(1) command. So, to get the free space on root file system you can use it as follows:
$ stat -f --format "%f" /
4238805
stat(1) command without --format option prints all the data about file system it can find out:
$ stat -f /
  File: "/"
    ID: b8a4e1f0a2aefb22 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 14066771   Free: 4238805    Available: 4234709
Inodes: Total: 3588096    Free: 2151591
This makes it in some way analogous to df(1) command. But, we are getting values in blocks, instead of kilobytes! You can get block size using %S format sequence, but that's it. So, some additional trickery is needed. One solution is to output arithmetic expression and evaluate it using bc(1) command, like this:
$ stat -f --format "%f * %S" / | bc
17362145280
Alternatively, it is also possible to use shell's arithmetic evaluation like this:
$ echo $((`stat -f --format "%f * %S" /`))17362145280
But, in both cases we are starting two process. In a first case the processes are stat(1) and bc(1), and in the second case it is a new subshell (for backtick) and stat(1). Note that this is the same as the solution with awk(1). But in case of awk(1) we are starting two more complex tools of which one, df(1), is more targeted to display value to a user than to be used in scripts. One additional advantage of a method using awk(1) might be portability, i.e. I'm df(1)/awk(1) combination is probably more common than stat(1)/bc(1) combination.

Anyway, the difference probably isn't so big with respect to performance, but obviously there is another way to do it, and it was interesting to pursue an alternative. 

Tuesday, January 22, 2013

Using ~ as a shortcut for home directory...

I just stumbled on the question why tilde (~) is used as a shortcut for home directory on Unix. It's very interesting since it never occurred to me to ask this question? :)

Then, there are lot more questions like that one on StackExchange, here is a selection of some interesting ones (to me at least):
And, for the end, here is why vi uses hjkl for cursor movement.

Saturday, January 28, 2012

Why we should study history...

Oh, it happened so many times! I stumbled upon it already while reading John Day's book titled Patterns in network architecture: A return to fundamentals. In short, at one point something is controversial and in next, it's regarded as a some kind of a rule that people passionately protect! One good example is 7-layer ISO/OSI Reference Model. When it was created it was problematic how many layers there should be, now, it is taken as something set in stone that there are 7-layers, while in reality it is dubious if this is a correct number. I'm certain that there are a large number of similar examples in every area you can think of. What this implies is that we have to always question the correctness of our current knowledge knowing that something might happen by chance, or politics of a certain time, and that ultimately hampers us from making further progress, maybe even clean start.

And today, I found this post. written by Rob Landley. It's ubeliviable! I'm using Unix/Linux for over 20 years now, always knowing there is a split between /bin, /sbin, /usr/sbin and /usr/bin and knowing why it is done so. But I realise now that, till today, I didn't actually know and, what's more, this is again an example of something that by accident becomes a law. What's more interesting is that not once I stumbled upon some heated discussion about file system layout (an example) in which there were proponents of this split with a simple argument that it is a Unix way of things! Boy, how wrong they are! :)

I'm copying this post here for a refence:

Usefull Unix commands, tips and tricks...

Have you seen this question on Hacker News? It's great and actually there are really some useful commands and tips for use of the command line more efficiently. Here I'll list some of them in case you want a quick glance, but be aware that something mentioned there might already be known to me and/or I decided that it's not so important and excluded it.

So, without further ado, here's my list of favorites:

Thursday, October 13, 2011

Dennis Ritchie died...

Well, another great figure of computing has prematurely left us, according to reports on the Internet. This one isn't so well known like Steve Jobs, but his work certainly matches the one done by Jobs, and in my humble opinion, even exceeds it. His "problem", sort to speak, is that he did everything in the core area of computer science, not in the consumer part, and he did majority of his work during the years when most people even didn't know that computers exists.

The guy is Dennis Ritchie, and he invented C programming language and also took important part in the development of Unix operating system. His influence was and is great. For example, Android smart phones today all run on top of Linux, which itself started as a Unix derivative. MS DOS was a very poor copy of Unix, and it was evident that it tried to copy Unix. Windows NT in part was also influenced by Unix. Not to mention MacOS X which, in its core, is Unix! And today's biggest businesses run their core services on Unix machines, not Windows.

The C programming language was, and still is, extremely influential. First, majority of today's operating systems are written in C, and all the other languages have ability to link with libraries written in C. There are numerous applications and libraries written in C. C is, in essence, lowest common denominator. Furthermore, we have today many languages which directly or indirectly borrow features from C. For a start C++ started as an extension to C. Which itself influenced many other object-oriented programming languages. C's influence can be traced also in all other non-OO languages.

All in all, I'm very sad that he passed away. RIP Dennis Ritchie.

About Me

scientist, consultant, security specialist, networking guy, system administrator, philosopher ;)

Blog Archive