Jekyll2017-04-02T23:20:40+00:00https://node-os.com/Node-OSThe first operating system powered by node.js and npmJsDayEs 20172017-03-24T18:45:39+00:002017-03-24T18:45:39+00:00https://node-os.com/blog/JsDayEs%202017<p>As you would already probably now I was candidate for the <a href="https://github.com/NodeOS/NodeOS/issues/278#issuecomment-273422673">JsDayEs 2017</a>… Ok, now it’s oficial: <a href="http://2017.jsday.es/es/#5740078466859008/99984001">I will talk about NodeOS</a> at one of the biggest Javascript events here at Spain :-D And crazy of me, I will do it in english, so it will be the perfect time to sumarize and finally have some documentation of NodeOS internal in english :-)</p>
<p>Now it’s time to clean-up a bit and pulish the project due this ocasion so we can be able to atract more collaborators, so I have two questions for you:</p>
<ol>
<li>what do you think it’s needed to do in the current NodeOS status to move it forward? From my mind, the only final tips for the final 1.0 release are the git client, fix nsh to allow interactive sessions, and probably update the website.</li>
<li>what do you want I talk about explicitly at the conference? :-) My initial intention is to talk about the story and architecture of NodeOS and what problems had surfaced during its development.</li>
</ol>
<p>Anything else? :-)</p>pirannaAs you would already probably now I was candidate for the JsDayEs 2017… Ok, now it’s oficial: I will talk about NodeOS at one of the biggest Javascript events here at Spain :-D And crazy of me, I will do it in english, so it will be the perfect time to sumarize and finally have some documentation of NodeOS internal in english :-) Now it’s time to clean-up a bit and pulish the project due this ocasion so we can be able to atract more collaborators, so I have two questions for you: what do you think it’s needed to do in the current NodeOS status to move it forward? From my mind, the only final tips for the final 1.0 release are the git client, fix nsh to allow interactive sessions, and probably update the website. what do you want I talk about explicitly at the conference? :-) My initial intention is to talk about the story and architecture of NodeOS and what problems had surfaced during its development. Anything else? :-)NodeOS 1.0.0-RC3 “refactor”… with a Bonus2017-01-24T14:14:30+00:002017-01-24T14:14:30+00:00https://node-os.com/blog/NodeOS%201.0.0-RC3%20%22refactor%22...%20with%20a%20Bonus<p>After some hiatus, there has been some work here on the project, and after some work… here is it the (in)famous <a href="https://github.com/NodeOS/NodeOS/releases/tag/v1.0.0-RC3">RC3</a> with the so much talked refactor :-D</p>
<p>It was known that NodeOS architecture was still too much dependent of the original centrilized design in part done due to the <a href="https://www.concursosoftwarelibre.org">CUSL</a>. This made it dificult to maintain because code was heavily cohexive, and the whole build process was painfully slow. After splittin it in several projects for each layer it was shown that this main project was almost empty having all the logic on each layer, so after some initial serious intention to port NodeOS to ARM once for all (but unsuccesful? More on this later), the conclusion was that it was needed to clean-up the code first. A lot. And so I did :-)</p>
<ul>
<li>Now all the logic regarding the generation of NodeOS for each platform is moved to the main NodeOS project itself, making each layer and sub-project to generate just simple and focused products that later are aggregated. Just plain ol’ school <a href="https://en.wikipedia.org/wiki/Unix_philosophy">UNIX filosophy</a>. For example, <a href="https://github.com/NodeOS/nodeos-barebones">nodeos-barebones</a> now just only generates a <code class="highlighter-rouge">.cpio.gz</code> initram to be embedded in the kernel (and the kernel itself, but it’s planned to be moved out too in the foresee future), without worrying about what particular NodeOS flavor will use it since it will be later customized, or <a href="https://github.com/NodeOS/nodeos-usersfs">nodeos-usersfs</a> now don’t have any dependency on the other layers, making it easier to understand how to create other custom users filesystems. It also convert and pack the different layers as the format needed by the target platform, being that a <em>tarfile</em> or an <code class="highlighter-rouge">ext2</code> filesystem or whatever, customizing them by adding or removing specific files if needed.</li>
<li><code class="highlighter-rouge">nodeos-rootfs</code> module has been finally deprecated and replaced by the <a href="https://github.com/NodeOS/nodeos-bootfs">nodeos-bootfs</a> tool, that generate the <code class="highlighter-rouge">/boot</code> partition or ISO image only when needed. It was there for legacy reasons before making each user to have their own root filesystem, but it was just a no-op in the cases where there was no need for a boot partition, so finally it was removed.</li>
<li>One important point is that each layer now has its own tests, that help to easily and sooner identify the problems that could arise by making each layer more independent, and in fact some of them has appear during the process of adding them (and a test is not good enough if it doesn’t show some bugs ;-) ). One of the biggest missing testing areas was to check that the prebuild release images of NodeOS where in fact bootable, that has been fixed both by creating a <a href="https://github.com/piranna/libblkid">wrapping module</a> on top of the <a href="https://www.kernel.org/pub/linux/utils/util-linux/v2.21/libblkid-docs">libblkid</a> library to be able to detect the partitions by their UUID instead of hardcoded device, and by showing the Syslinux bootloader prompt for just 0.1 seconds so the test can be able to detect and hook on it and append <code class="highlighter-rouge">console=ttyS0</code> to the kernel command line, so now it’s not only possible to capture its output and control the console but also the users can move the partitions around and add their own boot arguments without problems. The wrapper for <code class="highlighter-rouge">libblkid</code> has not be done without problems, due to the lack of documentation of how internally it works (at first I was not interested on compile it but just only to know how it detected the partitions UUIDs, because <a href="https://github.com/karelzak/util-linux">util-linux</a> is a huge monolithic library) and later due to a lot of legacy code that prevented to compile it easily with recent tools (I needed to <a href="https://github.com/piranna/libblkid/blob/ffa38b43d8247782608d42b97308c840488e9160/static/ensure_deps.sh#L47-L55">hack its code</a> to make it compile with <a href="https://github.com/nodejs/node-gyp">node-gyp</a>…).</li>
<li>Since now each layer is more independent and has their own tests it makes sense to also generate their own <code class="highlighter-rouge">prebuild</code> images automatically, and the fact is that this has helped to improve the NodeOS build time <strong>A LOT</strong>. By (ab)using the <a href="https://github.com/mafintosh/prebuild-install">prebuild-install</a> module now they checks if there’s available a prebuild image for the target platform, and if so then it’s used instead of download its source code and compile it, and to don’t need to download useless build dependencies for them, I’ve also created the <a href="https://github.com/piranna/buildDependencies">buildDependencies</a> module to download them for you in case they are needed by reusing the <code class="highlighter-rouge">devDependencies</code> field. So, do you remember the infamous step of <em>pick some microwave pop-corn and go to see a movie</em> from the <a href="https://github.com/NodeOS/nodeos#build-nodeos-in-five-steps">NodeOS build instructions</a>? Well… now with a decent internet connection and a somewhat fast laptop, maybe you’ll not be able to finish your cup of coffee while using the <code class="highlighter-rouge">BigRedButton</code> script to build and test <em>all</em> the NodeOS supported architectures and platforms <em>at once</em>. Yes, so fast is it now :sweat_smile:</li>
<li><code class="highlighter-rouge">nodeos-cross-toolchain</code> and the NodeOS layers build process now accept arguments following the QEmu scheme, and their generated products have been changued to follow them too. This has the side effect that now the <code class="highlighter-rouge">$PLATFORM</code> environment variable only define the format of the final products and has nothing to do with the architecture details. It also makes it easier to port to new platforms and architectures and to test them too by having a clear build matrix to follow. It keeps to convert the usage of specific “generic” CPUs identifiers to the more abstract Node.js architecture, so it would not be needed to be translated them several times but just once.</li>
<li><code class="highlighter-rouge">buildDependencies</code> or <code class="highlighter-rouge">nodeos-bootfs</code> have not been the only modules created for this RC. <a href="https://github.com/piranna/tar2ext">tar2ext</a> has been created to generate partition images for <code class="highlighter-rouge">usersfs</code> in a similar way to how <a href="https://github.com/piranna/cpio2tar">cpio2tar</a> works, and <code class="highlighter-rouge">nodeos-mount-filesystems</code> has been totally rewritten and splitted in the modules <a href="https://github.com/piranna/nodeos-boot-singleUser">nodeos-boot-singleUser</a>, <a href="https://github.com/piranna/nodeos-boot-singleUserMount">nodeos-boot-singleUserMount</a> and <a href="https://github.com/NodeOS/nodeos-boot-multiUser">nodeos-boot-multiUser</a> to improve flexibility in environments where it’s only desired <a href="https://github.com/NodeOS/NodeOS/issues/273">to exec a single Node.js app</a> instead of hosting a full multi-user environment. Not only that, that to ensure the single apps also run on an isolated environment similar to the one of the full NodeOS experience, the mechanism to create them has been put in the new <a href="https://github.com/piranna/jocker">jocker</a>, <em>the Javascript Docker</em> module :-P By the moment it’s just an augmentated <code class="highlighter-rouge">chroot</code>, but being on an independent module would help to move forward to use real LXC containers in the future in a simple way.</li>
</ul>
<p>As you can see there has been a lot of work here :-D But there’s still a lot to do. For example, <a href="https://github.com/piranna/buho">buho</a> module needs to be improved to be able to check on dependencies upgrading to update them too in a similar way to how <a href="https://greenkeeper.io/">Greenkeeper</a> works (maybe we could integrate it?), or Node.js itself it’s stalled on the 6.x branch due to <a href="https://github.com/nodejs/node/issues/9707">cross-compilation problems</a> on the version of the v8 Javascript engine included in Node.js 7.x, or the need to update <a href="https://github.com/mafintosh/fuse-bindings">fuse-bindings</a> to make use of <a href="https://github.com/mafintosh/fuse-bindings/issues/35">FUSE 3.0.0</a> or support several standard libraries on the <code class="highlighter-rouge">prebuild</code> images (<code class="highlighter-rouge">glibc</code> and <code class="highlighter-rouge">musl</code>) publishing releases of both, but definitelly the most important things that need to be fixed is the support to exec interactive console apps in <code class="highlighter-rouge">nsh</code> and the removal of the several projects forks by getting them to be merged upstream, so let’s start to give them some pressure ;-) Oh, and as always, improve the documentation of the different sub-projects, of course…</p>
<p>Regarding ARM, now both <code class="highlighter-rouge">nodeos-cross-toolchain</code> and <code class="highlighter-rouge">nodeos-nodejs</code> support them and host prebuild images for Raspberry Pi 2, but NodeOS itself can be tested due to problems with QEmu, seems regarding to the loading address of the kernel image. It needs to check if they run in fact on real hardware to confirm that’s a QEmu problem, but until that it has been added support for the Raspberry Pi original because it now has been added support for the board (also untested for the same reasons), and plan to add support for the <code class="highlighter-rouge">versalitepb</code> board since it’s the default one for ARM machines on QEmu and one year ago I was able to boot NodeOS on it (althought getting a kernel panic when executing the <code class="highlighter-rouge">/init</code> binary), so at least let’s hope to be able to tests the ARM architecture too :-)</p>
<p>And now, the…</p>
<h1 id="bonus">Bonus</h1>
<p>The RC3 was published last week, but I have been really busy this week and also caught a cold so my mind was not in the perfect state to concentrate and write this essay, so since I had back in my mind the feeling that I almost got something “interesting” to work, I employed the few spare time I got to work on it a little bit each time, and finally this morning I got it:</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code><span class="gp">piranna@slimbook-C16B:~/Proyectos/NodeOS$ </span>npm run docker
<span class="gp">> </span>NodeOS@1.0.0-RC3 docker /home/piranna/Proyectos/NodeOS
<span class="gp">> </span>scripts/docker
mount procfs: Resource busy
Hello! I<span class="s1">'m a user init script :-)
Welcome to NodeOS!: username: nodeos
· : password:
~ >
</span></code></pre>
</div>
<p>Yes, that’s it: NodeOS fully booting inside Docker!!! :-D The problem with <a href="https://github.com/NodeOS/nodeos-usersfs">usersfs</a> has been solved thanks to the refactor by being able now to convert the generated tarfile to a Docker Volume and assigning it directly to <code class="highlighter-rouge">/tmp</code> inside the NodeOS filesystem hierarchy, and it’s configured to support <code class="highlighter-rouge">FUSE</code> filesystems (needed by <a href="https://github.com/piranna/ExclFS">ExclFS</a>) and <a href="https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt">OverlayFS</a> (needed by the by-user root filesystem). It’s fairly experimental since I needed to tweak some things, for example I needed to change the Docker storage driver to <code class="highlighter-rouge">overlay2</code> because the default <code class="highlighter-rouge">aufs</code> one was freezing my machine needing to reboot, and that makes it impractical for debugging it and make it work (maybe now that’s fixed it could work but it’s untested, YMMV). To be honest I’ve needed to do some tricks like disable the usage of <a href="https://github.com/NodeOS/node-bin-getty">node-bin-getty</a> to manage the console because I was not being able to make it owner of the console (maybe I’m doing something wrong?) and exec directly <a href="https://www.npmjs.com/package/logon">logon</a> and also seems to be there a “little” problem with the user ID:</p>
<div class="language-sh highlighter-rouge"><pre class="highlight"><code>~ > pstree
init
├── exclfs
├─┬ nsh
│ └── pstree
└── nodeos-reverse-
~ > ls proc/
<span class="o">[</span> <span class="s1">'1'</span>,
<span class="s1">'110'</span>,
<span class="s1">'16'</span>,
<span class="s1">'45'</span>,
<span class="s1">'47'</span>,
<span class="s1">'acpi'</span>,
...
</code></pre>
</div>
<p>As we can see here, although we login with the <code class="highlighter-rouge">nodeos</code> account we are having the UID <code class="highlighter-rouge">0</code> and seeing the administrative processes, but at least it works to show how much few processes with administrative permissions are needed for a fully working system, that’s one of the purposses of NodeOS :-D I’m not sure if it could be a problem on NodeOS (I have checked the files directly on the layers managed by Docker and they have the correct UIDs and GIDs) or about Docker ignoring the UIDs and using just only <code class="highlighter-rouge">root</code> for everything because the LXC containers already isolate the processes from the underlying system so by design it’s supposed they are given the least possible permissions (I didn’t found <em>any</em> example that execute a process not as <code class="highlighter-rouge">root</code> inside a Docker instance so far…). In any case, that’s a huge milestone and promise some very interesting things for the future of NodeOS ;-) Next step, <a href="http://vagga.readthedocs.io">vagga</a> ;-)</p>pirannaAfter some hiatus, there has been some work here on the project, and after some work… here is it the (in)famous RC3 with the so much talked refactor :-D It was known that NodeOS architecture was still too much dependent of the original centrilized design in part done due to the CUSL. This made it dificult to maintain because code was heavily cohexive, and the whole build process was painfully slow. After splittin it in several projects for each layer it was shown that this main project was almost empty having all the logic on each layer, so after some initial serious intention to port NodeOS to ARM once for all (but unsuccesful? More on this later), the conclusion was that it was needed to clean-up the code first. A lot. And so I did :-) Now all the logic regarding the generation of NodeOS for each platform is moved to the main NodeOS project itself, making each layer and sub-project to generate just simple and focused products that later are aggregated. Just plain ol’ school UNIX filosophy. For example, nodeos-barebones now just only generates a .cpio.gz initram to be embedded in the kernel (and the kernel itself, but it’s planned to be moved out too in the foresee future), without worrying about what particular NodeOS flavor will use it since it will be later customized, or nodeos-usersfs now don’t have any dependency on the other layers, making it easier to understand how to create other custom users filesystems. It also convert and pack the different layers as the format needed by the target platform, being that a tarfile or an ext2 filesystem or whatever, customizing them by adding or removing specific files if needed. nodeos-rootfs module has been finally deprecated and replaced by the nodeos-bootfs tool, that generate the /boot partition or ISO image only when needed. It was there for legacy reasons before making each user to have their own root filesystem, but it was just a no-op in the cases where there was no need for a boot partition, so finally it was removed. One important point is that each layer now has its own tests, that help to easily and sooner identify the problems that could arise by making each layer more independent, and in fact some of them has appear during the process of adding them (and a test is not good enough if it doesn’t show some bugs ;-) ). One of the biggest missing testing areas was to check that the prebuild release images of NodeOS where in fact bootable, that has been fixed both by creating a wrapping module on top of the libblkid library to be able to detect the partitions by their UUID instead of hardcoded device, and by showing the Syslinux bootloader prompt for just 0.1 seconds so the test can be able to detect and hook on it and append console=ttyS0 to the kernel command line, so now it’s not only possible to capture its output and control the console but also the users can move the partitions around and add their own boot arguments without problems. The wrapper for libblkid has not be done without problems, due to the lack of documentation of how internally it works (at first I was not interested on compile it but just only to know how it detected the partitions UUIDs, because util-linux is a huge monolithic library) and later due to a lot of legacy code that prevented to compile it easily with recent tools (I needed to hack its code to make it compile with node-gyp…). Since now each layer is more independent and has their own tests it makes sense to also generate their own prebuild images automatically, and the fact is that this has helped to improve the NodeOS build time A LOT. By (ab)using the prebuild-install module now they checks if there’s available a prebuild image for the target platform, and if so then it’s used instead of download its source code and compile it, and to don’t need to download useless build dependencies for them, I’ve also created the buildDependencies module to download them for you in case they are needed by reusing the devDependencies field. So, do you remember the infamous step of pick some microwave pop-corn and go to see a movie from the NodeOS build instructions? Well… now with a decent internet connection and a somewhat fast laptop, maybe you’ll not be able to finish your cup of coffee while using the BigRedButton script to build and test all the NodeOS supported architectures and platforms at once. Yes, so fast is it now :sweat_smile: nodeos-cross-toolchain and the NodeOS layers build process now accept arguments following the QEmu scheme, and their generated products have been changued to follow them too. This has the side effect that now the $PLATFORM environment variable only define the format of the final products and has nothing to do with the architecture details. It also makes it easier to port to new platforms and architectures and to test them too by having a clear build matrix to follow. It keeps to convert the usage of specific “generic” CPUs identifiers to the more abstract Node.js architecture, so it would not be needed to be translated them several times but just once. buildDependencies or nodeos-bootfs have not been the only modules created for this RC. tar2ext has been created to generate partition images for usersfs in a similar way to how cpio2tar works, and nodeos-mount-filesystems has been totally rewritten and splitted in the modules nodeos-boot-singleUser, nodeos-boot-singleUserMount and nodeos-boot-multiUser to improve flexibility in environments where it’s only desired to exec a single Node.js app instead of hosting a full multi-user environment. Not only that, that to ensure the single apps also run on an isolated environment similar to the one of the full NodeOS experience, the mechanism to create them has been put in the new jocker, the Javascript Docker module :-P By the moment it’s just an augmentated chroot, but being on an independent module would help to move forward to use real LXC containers in the future in a simple way. As you can see there has been a lot of work here :-D But there’s still a lot to do. For example, buho module needs to be improved to be able to check on dependencies upgrading to update them too in a similar way to how Greenkeeper works (maybe we could integrate it?), or Node.js itself it’s stalled on the 6.x branch due to cross-compilation problems on the version of the v8 Javascript engine included in Node.js 7.x, or the need to update fuse-bindings to make use of FUSE 3.0.0 or support several standard libraries on the prebuild images (glibc and musl) publishing releases of both, but definitelly the most important things that need to be fixed is the support to exec interactive console apps in nsh and the removal of the several projects forks by getting them to be merged upstream, so let’s start to give them some pressure ;-) Oh, and as always, improve the documentation of the different sub-projects, of course… Regarding ARM, now both nodeos-cross-toolchain and nodeos-nodejs support them and host prebuild images for Raspberry Pi 2, but NodeOS itself can be tested due to problems with QEmu, seems regarding to the loading address of the kernel image. It needs to check if they run in fact on real hardware to confirm that’s a QEmu problem, but until that it has been added support for the Raspberry Pi original because it now has been added support for the board (also untested for the same reasons), and plan to add support for the versalitepb board since it’s the default one for ARM machines on QEmu and one year ago I was able to boot NodeOS on it (althought getting a kernel panic when executing the /init binary), so at least let’s hope to be able to tests the ARM architecture too :-) And now, the… Bonus The RC3 was published last week, but I have been really busy this week and also caught a cold so my mind was not in the perfect state to concentrate and write this essay, so since I had back in my mind the feeling that I almost got something “interesting” to work, I employed the few spare time I got to work on it a little bit each time, and finally this morning I got it: piranna@slimbook-C16B:~/Proyectos/NodeOS$ npm run docker > NodeOS@1.0.0-RC3 docker /home/piranna/Proyectos/NodeOS > scripts/docker mount procfs: Resource busy Hello! I'm a user init script :-) Welcome to NodeOS!: username: nodeos · : password: ~ > Yes, that’s it: NodeOS fully booting inside Docker!!! :-D The problem with usersfs has been solved thanks to the refactor by being able now to convert the generated tarfile to a Docker Volume and assigning it directly to /tmp inside the NodeOS filesystem hierarchy, and it’s configured to support FUSE filesystems (needed by ExclFS) and OverlayFS (needed by the by-user root filesystem). It’s fairly experimental since I needed to tweak some things, for example I needed to change the Docker storage driver to overlay2 because the default aufs one was freezing my machine needing to reboot, and that makes it impractical for debugging it and make it work (maybe now that’s fixed it could work but it’s untested, YMMV). To be honest I’ve needed to do some tricks like disable the usage of node-bin-getty to manage the console because I was not being able to make it owner of the console (maybe I’m doing something wrong?) and exec directly logon and also seems to be there a “little” problem with the user ID: ~ > pstree init ├── exclfs ├─┬ nsh │ └── pstree └── nodeos-reverse- ~ > ls proc/ [ '1', '110', '16', '45', '47', 'acpi', ... As we can see here, although we login with the nodeos account we are having the UID 0 and seeing the administrative processes, but at least it works to show how much few processes with administrative permissions are needed for a fully working system, that’s one of the purposses of NodeOS :-D I’m not sure if it could be a problem on NodeOS (I have checked the files directly on the layers managed by Docker and they have the correct UIDs and GIDs) or about Docker ignoring the UIDs and using just only root for everything because the LXC containers already isolate the processes from the underlying system so by design it’s supposed they are given the least possible permissions (I didn’t found any example that execute a process not as root inside a Docker instance so far…). In any case, that’s a huge milestone and promise some very interesting things for the future of NodeOS ;-) Next step, vagga ;-)New organization scheme2016-08-14T18:29:47+00:002016-08-14T18:29:47+00:00https://node-os.com/blog/New%20organization%20scheme<p>To make the NodeOS organization easier to manage I’ve removed the teams and now will continue with fine-grain access control. Now instead of having direct access to the repo, anybody that wants to contribute to NodeOS I can be able to add him to the NodeOS organization and assign him (or her) the corresponding issue to work with, so there’s no duplicated efforts. Due to that, they’ll need to create a pull-request that will be merged with the reference repo on the NodeOS organization, and also you can be able to create new repos without needing to ask me first (as happened with the <a href="https://github.com/NodeOS/docs">docs</a> repo). I will be able to create new teams for specific repos like the website that work together and make sense to have access to all of them at once, but in general terms will not be anymore the “all or nothing” access control of before.</p>
<p>I will continue to use directly the NodeOS organization repos until 1.0 version, after that I’ll protect the <code class="highlighter-rouge">master</code> branch of all of them and only it will be able to push and merge new commits the <a href="https://github.com/NodeOS-bot">NodeOS-bot</a> account managed by the CI server.</p>pirannaTo make the NodeOS organization easier to manage I’ve removed the teams and now will continue with fine-grain access control. Now instead of having direct access to the repo, anybody that wants to contribute to NodeOS I can be able to add him to the NodeOS organization and assign him (or her) the corresponding issue to work with, so there’s no duplicated efforts. Due to that, they’ll need to create a pull-request that will be merged with the reference repo on the NodeOS organization, and also you can be able to create new repos without needing to ask me first (as happened with the docs repo). I will be able to create new teams for specific repos like the website that work together and make sense to have access to all of them at once, but in general terms will not be anymore the “all or nothing” access control of before.How to get more contributors?2016-08-04T09:31:59+00:002016-08-04T09:31:59+00:00https://node-os.com/blog/How%20to%20get%20more%20contributors<p>I have been thinking about this topic since a long time ago but didn’t got yet a clear solution. It seems NodeOS has got some interest to be used on production or it’s already being used for it, but for being this possible in the long term it would need to be dedicated some more time and not being only a “hobby” done in our spare time both for maintenance and development of new features (NodeOS has became a really big project with a lot of dependencies that need to be maintained, although I’m trying to automate these tasks the most I can), and since we are several here that would be interested on getting some revenues, I think it’s time to ask the big question: how to monetize NodeOS, so I’ve open this issue to discuss for alternatives and try to find solutions for this.</p>
<p>The main problem I think we have are:</p>
<ul>
<li>How to measure contributions</li>
<li>There are collaborators from around the world in different timezones</li>
<li>There are collaborators that don’t participate anymore</li>
</ul>
<p>This makes it difficult to define a model fair for everybody beyond to use BountySource where anybody gets payed for the issues that he solve, but that doesn’t give promotion of NodeOS at big scale. It would be possible to create a company/association (I think Germany laws are easy for this, maybe @michielbdejong can give us some advice here, could you? :-) ) that give support, consultancy and expertise about NodeOS, but here the problem is how don’t give the feel that NodeOS is owned by that company.</p>
<p>A NodeOS Foundation would be the coolest option by putting the project in the first place and maybe would make it easier to get some sponsors and assists to events representing to NodeOS and not only just as individuals (as I have done on the keynotes and championships, but try to give on them the attributions to @groundwater for starting the project and the other collaborators as much as I can), but here we come again the the first problem: how to distribute the money in a fair way.</p>
<p>Another option, maybe the better, is to forget by the moment about the beneficts and create a non-proffit organization just to have some identity and to promote NodeOS itself, try to get some sponsors for travelling to conferences and little more (at this moment we don’t have any cost on servers or similar, only the website domain), and if things go well, maybe promote the organization to a foundation and dedicate those sponsor funds to pay a developer for one or two months working full or part-time or by objectives. I admit from start that I would candidate for that if we get to that situation (just to be clear and honest), but although I’m the most active one here and has a somewhat mind-map of how I want NodeOS to be, I’m not propossing this to “get all the money” but instead I’m very concerned and worried about how to make others to benefict for their work on NodeOS.</p>
<p>Also, there’s the big problem of promote NodeOS and nock some doors to get sponsors of the project and financiation… I think Node.js Foundation and NPM org would be the obvious candidate to ask for, but we would need to show we are not just some kids playing here, so adding native git support for npm with a pure javascript library instead of relaying on an external git cli command would be a really good presentation letter. Later we would consider other big players like Mozilla Foundation if we get a good integration with B2G OS (I know it’s community driven, but sure they are not unaware what’s happening on their community ;-) ) and if we got luck we could also ask to the Linux Foundation as an example of how powerful and easy to use Linux has became in the last years.</p>
<p>Opinions? Suggestions? There are some list here on GitHub about finatial mechanism for Open Source projects, it would be a good idea to reference them here too.</p>
<p>/cc @luii, @5paceManSpiff, @SpaceboyRoss01, @kskarthik, @Coretool, @lapineige, @lissyx, @Ryuno-Ki, @khanshakeeb</p>pirannaI have been thinking about this topic since a long time ago but didn’t got yet a clear solution. It seems NodeOS has got some interest to be used on production or it’s already being used for it, but for being this possible in the long term it would need to be dedicated some more time and not being only a “hobby” done in our spare time both for maintenance and development of new features (NodeOS has became a really big project with a lot of dependencies that need to be maintained, although I’m trying to automate these tasks the most I can), and since we are several here that would be interested on getting some revenues, I think it’s time to ask the big question: how to monetize NodeOS, so I’ve open this issue to discuss for alternatives and try to find solutions for this.Refresh the Website!2016-06-26T00:42:03+00:002016-06-26T00:42:03+00:00https://node-os.com/blog/Refresh%20the%20Website!<p>Hi! Are you a front-end web developer? Now is your chance for you and your buddies to contribute to NodeOS (without having to decipher underlying technology in the build process; double win!)</p>
<p>We’re redesigning our website and would love to have the talent’s of all the developers combined! The first step, of course, would be to design a new page so let’s use this thread and create a new idea. You may propose:</p>
<ul>
<li>Theme idea’s</li>
<li>Template ideas (you can sketch, or paste screenshots)</li>
<li>Web frameworks (like bootstrap) that we should (or shouldn’t) use, and why</li>
</ul>
<p>Looking forward to getting this going with you all! Tell all your friends!</p>lite20Hi! Are you a front-end web developer? Now is your chance for you and your buddies to contribute to NodeOS (without having to decipher underlying technology in the build process; double win!)1.0.0-RC22016-06-25T23:44:56+00:002016-06-25T23:44:56+00:00https://node-os.com/blog/1.0.0-RC2<p>Just exactly one month after my degree thesis (with distinction, f*ck yeah ;-) ) and with a lot of nights without sleep, we have available <a href="https://github.com/NodeOS/NodeOS/releases/tag/v1.0.0-RC2">NodeOS 1.0.0-RC2</a>.</p>
<p>There has been some problems lately with the packages and the build process, but they are due to the heavy changes on the architecture that I hope they are going to be better for the future. The main one, is the split of the different layers on independent, isolated and autonomous projects that are themselves published on NPM, but also (and that’s the important part) they have their own tests <a href="https://semaphoreci.com/nodeos">passing on SemaphoreCI</a> and configured to work with <a href="http://greenkeeper.io">GreenKeeper</a>, so when we’ll move to 1.0.0 final we’ll be able to work on a real Rolling Release distro, and also there have been suggestion to <a href="https://github.com/NodeOS/NodeOS/issues/247">include an autoupdater</a> on NodeOS itself. As an implementation detail, these layers and submodules (like <a href="https://github.com/NodeOS/qemu">qemu</a>, that it’s now <a href="https://www.npmjs.com/package/qemu">published on npm</a>) use extensively the <a href="https://github.com/mafintosh/prebuild-install">prebuild-install</a> module to distribute prebuild images of their components, so no more pop-corn & movies to build NodeOS but just only a coffee. Yes, build process it’s <em>blasingly fast</em>, and there’s also a lot of room for improvement so we could end up blurrying the line between install and compile times by merging both processes ;-)</p>
<p>Other important change due to the increasing number of installing problems is the movement to npm3 and deprecation of npm2, so now build process is deterministic, faster and use less memory (also in NodeOS disk images, that have decreased ;-) ). Only drawback is that since npm3 considers the <code class="highlighter-rouge">install</code> process a one-time execution it’s needed to split the install of the dependencies (<code class="highlighter-rouge">npm install</code>) and the generation of the NodeOS image (<code class="highlighter-rouge">npm run build</code>), but this also allow better flexibility and also remove the need of network connection when building the system since all the modules included on the NodeOS images has been set as dependencies of the NodeOS module itself, so they are getting cached for later.</p>
<p>But probably the most important change is Node.js, because not only it has got <a href="https://www.npmjs.com/package/nodejs">it’s own npm package</a> downloading prebuild binaries based on musl (maybe we should propose to Node.js org to host and distribute them side-by-side with their glibc ones?), but also it has been upgraded to bleeding edge 6.2.2! :-D There has been some problems because <code class="highlighter-rouge">/proc</code> was not being mounted and <a href="https://github.com/nodejs/node/issues/7175#issuecomment-227170756">it’s needed by musl</a>, but thanks to it I give the chance to do some improvements to <code class="highlighter-rouge">nodeos-init</code> and now it’s easier to use and robust, cleanly clossing the system itself on shut-down :-)</p>
<p>Let’s more to 1.0.0! :-D</p>pirannaJust exactly one month after my degree thesis (with distinction, f*ck yeah ;-) ) and with a lot of nights without sleep, we have available NodeOS 1.0.0-RC2.Keynote at UC3M Linux Users Group2016-04-14T19:56:21+00:002016-04-14T19:56:21+00:00https://node-os.com/blog/Keynote%20at%20UC3M%20Linux%20Users%20Group<p>As some of you would already know if you follow us on <a href="https://twitter.com/TheNodeOS">Twitter</a>, last Tuesday I was giving a keynote about NodeOS on the <a href="http://www.gul.es/">Charles Third University Linux Users Group</a>, and finally I’ve got the <a href="https://drive.google.com/file/d/0B9fEpRL8W1SPZUFkeGNDYk1RSlE/view?usp=sharing">recording</a> of the keynote ready to be downloaded :-)</p>
<p>And in other news… :-P I have been fiddleing the last days with <a href="https://github.com/piranna/coverdeeps">coverdeeps</a>, a little tool to check the global test coverage of a project and all its dependencies, since they are important (or more!) than the main project. It’s currently limited to project that are already published on the npm registry and using <a href="https://coveralls.io/">coveralls.io</a> for test coverage publishing so it’s really pesimist about the results, but it’s a good beginning the NodeOS objective to being an OS (the first one?) with a perfect score of <a href="https://github.com/NodeOS/NodeOS/issues/75">100% test coverage</a> :-)</p>pirannaAs some of you would already know if you follow us on Twitter, last Tuesday I was giving a keynote about NodeOS on the Charles Third University Linux Users Group, and finally I’ve got the recording of the keynote ready to be downloaded :-)A Cool Thought2016-04-12T06:21:48+00:002016-04-12T06:21:48+00:00https://node-os.com/blog/A%20Cool%20Thought<p>I just wanted to bring up something cool I was thinking about today. NodeOS and Android share as many similarities as British vs American English.</p>
<p>Both systems are based off the Linux kernel. Android actually just uses kernel modules for all hardware interaction and uses the JNI to interact. In the same way, NodeOS uses v8’s bridge to the native interface to access kernel operations.</p>
<p>There are also deep relationships between the two languages. Aside from the notable similar , code structure, and fact that they’re both , both projects were worked on by the same developers. The same developers who created the Java Hotspot VM, worked on V8 too.</p>
<p>I find the similarities quite interesting. We do of course have differences. NodeOS is arguably more portable, capable of supporting more technology with less work (less layers between userspace and kernel modules).</p>
<p>I could keep going but eh, it’s all cool!</p>lite20I just wanted to bring up something cool I was thinking about today. NodeOS and Android share as many similarities as British vs American English.Prebuilds, again2016-04-08T20:05:43+00:002016-04-08T20:05:43+00:00https://node-os.com/blog/Prebuilds,%20again<p>In the last weeks it could seems there have been too few movement in the project due to several reasons: NodeOS core is stabilized after the <a href="https://github.com/NodeOS/NodeOS/issues/181">RC1</a>, and both me and other members of the project we have been busy with work and classes (I hope to read my thesis in May! :-D ). At the end, we are working on NodeOS on our spare time and we have some real-life responsabilities…</p>
<p>The point is that we have been banging our head against some walls while moving towards the 1.0 version. For example, @tylerl0706 has been getting some non-reproducible issues while trying to fix the pending bits to port <a href="http://www.nodegit.org/">nodegit</a> so we can have a working git client on NodeOS (needed to use git repos with npm). In my case, although I got some good progresses <a href="https://github.com/piranna/nsh">reimplementing nsh as a bash-languaje interpreter</a> running on a single process (that would lead in the future to write a <em>bash-2-javascript</em> transpiler) instead of just executing the user commands, the fact is that I’ve <a href="https://github.com/nodejs/node/issues/5574">found a bug on Node.js</a> where stdin stream is not correctly processed and makes it unusable and the alternative is to not be able to auto-detect when it’s running on an interactive shell, and also trying to fix <a href="https://github.com/kevva/download">download</a> module to <a href="https://github.com/kevva/download/issues/88">decompress on real time</a> instead of hold the files in ram (making it impossible to download NodeOS dependencies on memory constrained systems) lead to some cryptic errors or random incomplete downloads without notification. Really frustrating :-( Good things are, building an OS a such a complex task that always there’s something to do, and if something it’s burning yourself you can try with another thing :-)</p>
<p>One recurrent issue people is having is that <code class="highlighter-rouge">nodeos-cross-toolchain</code> module <a href="https://github.com/NodeOS/NodeOS/issues/218">dissapears</a> when using latest Node.js 5. The reason is that it includes npm 3, that has changed the organization of the <code class="highlighter-rouge">node_modules</code> folder to being maximally flat, deleting its folder on the way. Until now the only solution is to still use Node.js 4 and npm 2, but it will someday gets support and it’s time to move on, so since it was almost autonomous I’ve finally converted it on an <a href="https://github.com/NodeOS/nodeos-cross-toolchain">independent npm module</a> thanks to a modified version of the use of <a href="https://github.com/piranna/prebuild">prebuild</a> to be able to use some <a href="https://github.com/NodeOS/nodeos-cross-toolchain/releases">prebuild releases</a> of general purpose binaries. This allow to use <code class="highlighter-rouge">nodeos-cross-toolchain</code> as a dependency of the NodeOS layers without needing to build them again and again, but also move forward to the npm 3 support and more important to allow to test it independtly and reduce NodeOS build time to the half (yes, the cross-compiler takes so much time as NodeOS itself…), so now instead of <a href="https://github.com/NodeOS/NodeOS#build-nodeos-in-three-steps">pick some microwave pop-corn and go to see a movie</a> it’s just a matter of see one chapter of your favorite soap opera :-P It has helped to reduce build time not only to convert the cross-toolchain on a precompiled npm module, but also I’ve created prebuild modules for <a href="https://github.com/NodeOS/genfatfs">genfatfs</a>, <a href="https://github.com/NodeOS/genext2fs">genext2fs</a> and <a href="https://github.com/NodeOS/qemu">qemu</a>. Yes, you’ve read it right: now you can use <a href="http://qemu.org">QEmu</a> as a <a href="https://github.com/NodeOS/NodeOS/blob/7cb4bb925986a97fb7ffd9e6fdf469b583cb56bd/package.json#L45">npm dependency</a>, no kidding :-D</p>
<p>But to be honest, upcoming npm 3 support what not the main reason to move cross-toolchain as an independent module, but NodeOS prebuild images. NodeOS build process has got to be so much complex and heavy weight that the CI server was getting random errors constantly (and in fact I’m thinking to move to another one due to this, but I’ve not found any other free one as fast… maybe is it time to give another opportunity to TravisCI?), so the only possible solution was to split the build process in several independent fragments and test them separately, tha obvious first one was the cross-toolchain since it only would need changes when a new version of GCC or Linux kernel arises. So after adding the dependencies and tuning-up the build scripts and fix some bugs on the <a href="https://github.com/piranna/publish-release">publish-release</a> module… finally we have <a href="https://github.com/NodeOS/NodeOS/releases">prebuild NodeOS images</a> again after 4 months of silence, just ready so people can test the upcoming 1.0 version :-D</p>
<p>Last but not least, another tiny bit that has been floating around since some time ago was the possibility to add an <a href="https://github.com/NodeOS/NodeOS/issues/177">administrator mode</a> that would allow to admin some system global configs like to allow to <code class="highlighter-rouge">logon</code> to <a href="https://github.com/piranna/logon/issues/3">create new users</a>, while at the same time don’t allow users to log in the system to prevent problems like the fact in WIndows most people (specially in domestic environments) access to the system with administrator permissions. It was easier to implement that what I though on a first time, just by <a href="https://github.com/NodeOS/nodeos-mount-filesystems/blob/7937600616a6987d472d0e85d9c4143ec9585c97/server.js#L285">forcing the user to a REPL</a> if system is booted with the <code class="highlighter-rouge">single</code> flag and don’t execute the users <code class="highlighter-rouge">init</code> processes (specially the one for <code class="highlighter-rouge">root</code>, that start <code class="highlighter-rouge">logon</code>). This also help me to find a bug on <a href="https://github.com/piranna/nodeos-init">nodeos-init</a> where it was trying to exec as a command any positional argument that it was being given so it though <code class="highlighter-rouge">single</code> was a command (when it’s not :-) ), so now it <a href="https://github.com/piranna/nodeos-init/commit/c23ce46be2ada9e1899b9129a4a162507c43ff55">detects</a> if the first one is an absolute path (since at boot the kernel doesn’t define itself the <code class="highlighter-rouge">$PATH</code> environment variable used to locate the binaries) and if not, it gives all the parameters as arguments for the default <code class="highlighter-rouge">/sbin/init</code> binary that mount the partitions and boot the system :-)</p>
<p>So as you can see, steady NodeOS is moving towards it’s 1.0 version! :-D Let’s hope the issues that have appear on its dependencies modules can be fixed soon and we can release something we can be proud of :-)</p>pirannaIn the last weeks it could seems there have been too few movement in the project due to several reasons: NodeOS core is stabilized after the RC1, and both me and other members of the project we have been busy with work and classes (I hope to read my thesis in May! :-D ). At the end, we are working on NodeOS on our spare time and we have some real-life responsabilities…Not a toy OS anymore2016-03-03T00:11:31+00:002016-03-03T00:11:31+00:00https://node-os.com/blog/Not%20a%20toy%20OS%20anymore<p>With the <a href="https://github.com/NodeOS/NodeOS/issues/181">release of the release candidate</a> it would seems that this repository has been somewhat quiet, but the fact is that there has been a lot of progress lately on other areas of NodeOS (and I’m not talking only about the new 700 stargazers or the repercussion on Twitter of the project and the publication of several <a href="https://www.toptal.com/nodejs/nodeos-the-javascript-based-operating-system">articles on Internet</a>… ;-) ) like my bachelor thesis or the rewrite of <a href="https://github.com/piranna/nsh">nsh</a> (still in progress, but when I get to rid the stdin problems it will be really awesome :-D ) and specially thanks to the work of @lite20, @Coretool and @formula1 here on the issues but also on projects like <a href="https://github.com/Coretool/noGUI">noGUI</a> (keep rocking guys!!! ;-) ).</p>
<p>But last February 29th (what a charismatic date ;-) ) something “magic” happened, that by the way while I’m writting this <a href="https://github.com/NodeOS/NodeOS/commit/050bb316986953726f661eb771fa3bb822890693#commitcomment-16460007">there are some spies</a> asking me for details ;-)</p>
<p>The suggestion of @Xe about <a href="https://github.com/NodeOS/NodeOS/issues/187#issuecomment-164025249">using an independent C-based init process</a> didn’t worked as expected but at least it helped to have an <a href="https://github.com/piranna/nodeos-init">init process</a> that shutdown gracefully the system when no more processes are running.</p>
<p>But I was thinking: Node.js can run on regular Linux systems, so <em>something</em> happens between starting the system <code class="highlighter-rouge">init</code> and executing Node.js that allow latest ones to run correctly that was not needed before. Maybe the kernel filesystems like <code class="highlighter-rouge">/proc</code> or more probably <code class="highlighter-rouge">/sys</code> (that I was not using before)? So trying to isolate it I made a copy of my Ubuntu <code class="highlighter-rouge">initrd.img</code> file and <a href="https://scaryreasoner.wordpress.com/2009/08/29/debugging-the-linux-boot-process/">included it</a> Node.js and its needed libraries. Later booted my computer using it and disabling my hard disk to force to keep into the initram and there I was able to run Node.js. Cool! Time to clean the <code class="highlighter-rouge">init</code> script. Mount root partition, NFS, parsing Linux command line… at the end I get only three suspects: the mounting of <code class="highlighter-rouge">/sys</code>, <code class="highlighter-rouge">/proc</code> and <code class="highlighter-rouge">/dev</code> filesystems, <em>of course</em>.</p>
<p>But I was afraid. What if I was wrong? What if by removing them Node.js <em>still runs</em> on the Ubuntu initram? Could be the problem be somewhere else? And ironically, that would have been the time I most enjoyed something doesn’t work :-P After checking it found the culprit: <code class="highlighter-rouge">/dev</code>, the system devices filesystem. WTF? What could have changed on v8 that makes uses of the system devices that’s missing and makes it consume the full CPU? Anyway I didn’t wanted to fix the Ubuntu initram but instead to make NodeOS to boot with newer versions by mounting <code class="highlighter-rouge">/dev</code> before executing Node.js, so now that I have an independent <code class="highlighter-rouge">init</code> process that would be a good candidate to do it. <a href="https://github.com/piranna/nodeos-init/commit/e60693419caf5f8740d2b1b6767e8dccd85bbf59">Just a couple of lines</a>, and check that everything was working. Ok, let’s go for the real deal. Use Node.js 0.12.10 (one of the closests to the 0.11.14 one used by NodeOS) as I was using on my tests with the Ubuntu initram? No, let’s go crazy, run before walk, fail with a lot of noise and use the latest 4.3.1 stable. And after 30 minutes compiling… <em>magic</em>:</p>
<p><img src="https://cloud.githubusercontent.com/assets/532414/13479054/a51aa6c0-e0d4-11e5-930b-aaa9e8d51470.jpg" alt="img-20160229-wa0003" />
NodeOS barebones layer. Using Node.js 4.3.1. <strong>For real</strong></p>
<p>I don’t have words to explain that moment beyond the fact my cat run scaried and my parents though I got insane :-P One year and a half looking back for a bug… that would turn mad to anybody :-P</p>
<p>After the exctatic moment and some well gained nice tea drinking (not pun intended, I still don’t sell hats :-P ), I needed to update all the NodeOS dependencies that use a compiled module due to the fact I was using outdated ones that were compatible with Node.js 0.11.14. Really a lot (8 of them just only on the <a href="https://github.com/NodeOS">NodeOS organization</a>…). It was greatly useful the previous work done by @heavyk (since I’ve never used <a href="https://github.com/nodejs/nan">nan</a> before), and after two days now we have NodeOS fully working with Node.js 4.3.1 and ready for the upcoming ones :-)</p>
<p>There are not prebuild images yet because the build process has got to be really huge and fat and the CI server is not capable to manage so much downloads and compiling steps. I’m thinking about moving to another more capable alternative, but there are also son discussions and <a href="https://github.com/NodeOS/NodeOS/pull/214">pull-requests</a> to remove some functionality from the build process and move it to <a href="https://github.com/NodeOS/nodeos-cross-toolchain">another independent projects</a> that could more easily be tested and generated and also get it faster to generate NodeOS itself by using prebuild binaries.</p>
<p>Anyway, the code is already available in master branch, and whatever direction the project carry on from now, keep this sentence in your mind: <strong>NodeOS is not a toy OS anymore</strong>, and it’s very capable to be used on production environments for real use cases from now on. F*ck yeah B-)</p>pirannaWith the release of the release candidate it would seems that this repository has been somewhat quiet, but the fact is that there has been a lot of progress lately on other areas of NodeOS (and I’m not talking only about the new 700 stargazers or the repercussion on Twitter of the project and the publication of several articles on Internet… ;-) ) like my bachelor thesis or the rewrite of nsh (still in progress, but when I get to rid the stdin problems it will be really awesome :-D ) and specially thanks to the work of @lite20, @Coretool and @formula1 here on the issues but also on projects like noGUI (keep rocking guys!!! ;-) ).