Failure at piloting external processes stdin and stdout using Ruby on Windows

I have always tried to be a polyglot programmer, and also poly-platform one.
I have always cherished the idea my softwares could run on any OS. My main development environments for most of my Ruby projects are Windows (XP and 7), *nix and Cygwin.

So far I always managed to implement cross-platforms solutions, even if they sometimes needed some dirty targeted if…then based on platforms or Ruby versions.

However today I ran into a wall.

I want to execute an external process, get its output in real time, and pilot it by writing to its input in the same time.
The goal is to mimic a real user’s behavior reading his screen and writing to his terminal in real time.

Simply put: I gave up on native Windows. Cygwin is then mandatory for this task 🙁
EDIT: I managed to find a solution: see my next blog post for details.
EDIT 2: I packaged the solution in a nice Ruby gem: ProcessPilot.

I first tried to use Ruby’s standard library or external gems to spawn the process and get access to its stdin and stdout.
After failing, I tried to do it manually, by forking and redirecting pipes between parent and child processes and piloting the child process from his parent. Still fails.
Then I thought about using files as redirections for stdout, stderr and stdin. This works perfectly for stdout and stderr, but no way for stdin. Even if the stdin file is empty, the process still mimics inputs (and I tested it even without Ruby – this is a Windows/cmd.exe behavior).

Here are the main points I considered:

  • Open3::popen3, IO::popen, Process::spawn, Kernel::open, systemu all behave the same way: you can’t get output in real time. You get the whole output once the process exits.
  • PTY::spawn is the perfect solution, but there is no implementation on Windows.
  • Native Ruby on windows does not have a fork implementation.
  • win32-process gem offers a fork implementation on Windows, but descriptors are broken by the fork. Therefore it is impossible to pilot the child process from its parent by redirecting its stdin and stdout to pipes.
  • A Windows command cannot receive its stdin input from a streamed file. The file has to contain all the lines to input at the beginning of the process run.

If any of my readers have better clues, please share them for the sake of everyone!

Ruby, Windows , , , , , , , , , , , , , , ,

How to access a GIT repository over authenticated HTTP

Hi

My second headache for today was brought by GIT. I have a simple Apache server providing a restricted access (basic HTTP authentication) on a GIT repository.

From my local terminal, I just wanted to access the repository using:
git clone http://myhost.com/path/to/repository/.git
It asks for credentials (both user name and password), and fails miserably with a 401 error after entering them.

Well the problem is solved by just specifying the user name in the URL:
git clone http://myuser@myhost.com/path/to/repository/.git
This way it asks for just the password, and works like a charm.

Seems like a bug to me :-/

Muriel

Git, Howto , , , , , , ,

How to make RewriteCond work with sub-directories needing HTTP authentication

I almost spent 10 minutes trying to figure out a title for this post, as the problem I stumbled upon today is not easy to describe 😉

In a basic Apache environment, I have a rewrite rule routing every incoming request to another listening port. However I want just 1 directory to not be rerouted. So I used this .htaccess configuration:

RewriteEngine on
RewriteCond %{HTTP_HOST} ^myhost.com$
RewriteCond %{REQUEST_URI} !^/dirnotrerouted
RewriteRule ^(.*)$ "http\:\/\/127\.0\.0\.1\:12000\/$1" [P,L]

Therefore every incoming request to http://myhost.com/whatever is redirected to http://myhost.com:12000/whatever and any access to http://myhost.com/dirnotrerouted/whatever is not rerouted. It works.

Now I want to add basic HTTP authentication to one of the directories inside dirnotrerouted. So I created a specific .htaccess file in the restricted directory:

AuthType Basic
AuthName "Restricted area"
AuthUserFile /path/to/.htpasswd
Require valid-user

And now comes the problem: accessing http://myhost.com/dirnotrerouted/restricted is now being rerouted to http://myhost.com:12000/dirnotrerouted/restricted. Problem does not occur for other directories.

And now the solution: specify a 401 error document in the .htaccess file.

ErrorDocument 401 "Unauthorized Access"

AuthType Basic
AuthName "Restricted area"
AuthUserFile /path/to/.htpasswd
Require valid-user

This way your directory is now accessible without being redirected, and is using HTTP authentication.

Muriel

Howto, Web development , , , , , , ,

How to install reCaptcha on your MediaWiki

Lately I just ran into heavy spamming among all my MediaWiki’s wikis.
It first began with plain anonymous spamming (I wanted to leave anonymous editing available), then I forced account creation, then spammers were automating new account registrations before updating existing pages and creating new ones with spam.

So I decided to do things properly and install the reCaptcha plugin for MediaWiki.

Apparently the problems you may encounter while doing so highly depends on your PHP and MediaWiki versions.
Here are mine:

  • PHP: 5.3.2
  • MediaWiki: 1.14.0 and 1.15.1
  1. Create a set of public/private keys using Google’s reCaptcha service.
  2. Follow instructions on Google’s help to install reCaptcha’s MediaWiki plugin. At the time of this writing, I used version 1.7.
  3. An additional step is needed to allow editing pages. Modify file extensions/recaptcha/ReCaptcha.php, by replacing the following line
    function confirmEdit( &$editPage, $newtext, $section ) {
    

    with this:

    function confirmEdit( $editPage, $newtext, $section ) {
    

    (Just remove the & character). This issue is covered in this ticket.

And here you go: up and running MediaWiki with reCaptcha. You can check one of my installations here.

Antispam, Howto, Web development , , , , ,

No more STATUS_ACCESS_VIOLATION on Cygwin under Windows7

I recently reinstalled Cygwin 1.7.9-1 on a Windows7 Ultimate edition (64bits), and like many many people, I ran into the problems of randomly thrown exceptions by xterm and alike processes.

For example when starting xterm:

       4 [main] xterm 7156 exception::handle: Exception: STATUS_ACCESS_VIOLATION
    1449 [main] xterm 7156 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       4 [main] xterm 7996 exception::handle: Exception: STATUS_ACCESS_VIOLATION
    1356 [main] xterm 7996 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       4 [main] xterm 5268 exception::handle: Exception: STATUS_ACCESS_VIOLATION
     647 [main] xterm 5268 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       5 [main] xterm 4316 exception::handle: Exception: STATUS_ACCESS_VIOLATION
     739 [main] xterm 4316 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       2 [main] xterm 2752 exception::handle: Exception: STATUS_ACCESS_VIOLATION
    1123 [main] xterm 2752 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       5 [main] xterm 3220 exception::handle: Exception: STATUS_ACCESS_VIOLATION
    1260 [main] xterm 3220 open_stackdumpfile: Dumping stack trace to xterm.exe.stackdump
       6 [main] xterm 6752 fork: child -1 - died waiting for longjmp before initialization, retry 0, exit code 0x600, errno 11
xterm: Error 29, errno 11: Resource temporarily unavailable
Reason: spawn: fork() failed

When googling, some people seem to solve it just by disabling Windows DEP. Well for me it did not work (errors were exactly the same with DEP exceptions for Cygwin processes, and even with DEP completely disabled).

I solved it by executing some Cygwin maintenance that should definitely be integrated in Cygwin’s setup, but is not as of today.
This maintenance task is named rebaseall: you have to first install it as a Cygwin package, and you can find a very useful readme in /usr/share/doc/Cygwin/rebase-X.Y.Z.README file, explaining the ins and outs of this process.

  1. Modify /bin/rebaseall script: This script needs a minor modification to not fail on 64bits MingW DLLs. Change its line 110 from

        sed -e '/cygwin1\.dll$/d' -e '/cyglsa.*\.dll$/d' -e 's/^/\//' >"$TmpFile"
    

    to

        sed -e '/cygwin1\.dll$/d' -e '/mingw\/bin.*\.dll$/d' -e '/cyglsa.*\.dll$/d' -e 's/^/\//' >"$TmpFile"
    
  2. Copy /bin/ash.exe to /usr/bin/ash.exe (eventually create the directory if it does not exist).
  3. Kill all Cygwin related processes. This also includes any xterm, bash, and even XWin.exe server.
  4. Execute /bin/ash.exe
  5. In the ash terminal, execute /bin/rebaseall. It should (and has to) execute without any errors, otherwise it will leave your Cygwin install in a very unstable state. Warnings are acceptable though.
    If it fails because of a used DLL, an executing Cygwin process, or anything, don’t panic: correct the problem (kill running Cygwin processes if any), and try again.
  6. Still in the ash terminal, execute /bin/peflagsall. This should also run without errors, just warnings.

Then you’re goot to go. No need to restart, you should have your Cygwin processes running without exceptions.

Cheers

Muriel

Cygwin, Howto , , , , , , , ,

How to make a customized Ruby on Rails server work on a hosted shared environment ?

Hi

Some of my servers are hosted on Planethoster (shared environment).
So far it has been a nice experience, except that their customer support’s technical competencies can vary a lot depending on who gets to help you there.

Regarding their Ruby on Rails setup, it uses Mongrel over Apache, and provide cPanel as an admin interface. I found out that many of their servers have different versions of Mongrel, Ruby on Rails, and Rubygems. This can be quite annoying all the more so as some of them have incompatible versions, hence the impossibility to sometimes create new rails projects from the cPanel, or even start servers. The problem becomes even more complicated when you try to install your own gems in your user space.

So here are my advices on how to deal with a correct Ruby on Rails setup there.
The example is taken from my own Redmine installation (which you can check here).

  1. Get an SSH access to your server. This is obvious, as otherwise you will have to deal with their crontab and scripts sent by FTP to launch your database migrations and other rails maintenance tasks.
  2. If you want to create your Rails application, follow those steps:
    1. First try to create the Rails application using the cPanel. If it fails because of non matching Rails version or another unknown error, don’t worry, we’ll create it manually.
    2. To create the Rails application manually, just use your SSH access, and try running your rails commands in the ~/rails_apps directory.There, you can see the real errors preventing the creation, and take corrective actions if needed: install the gems you want (even a complete new Rails version if needed), modify your GEM_PATH environment variable if needed (in this case you’ll have to remember the working value to set it later in further points).Although I did not try it, I think you can also install and use other Mongrel versions in your user space.
  3. If you want to set up an existing Rails application in this environment (the ~/rails_apps directory and the cPanel) for the first time, follow these steps:
    1. First try to create a new empty Rails application using the cPanel. If it fails because of non matching Rails version or another unknown error, don’t worry, set it up manually by following step 2.B.
    2. Then use your SSH access to copy all your Rails application files in the created directory.
  4. Create a config file setting the environment for your own gems. This is necessary if you want your own gems (saved in your user space) to be used. For example here is the file I created:
    require 'rubygems'
    ENV['GEM_PATH'] = "/home/username/ruby/gems:/home/username/.gem/ruby/1.8:/usr/lib/ruby/gems/1.8"
    Gem.clear_paths
    

    This file will be used to enforce the setup of your gems in any Ruby environment, as it is not always done by default (crontab environments or cPanel starting Mongrel may not set it); this way you will be on the safe side.

  5. Require your environment file in your Rails’ Rakefile. This is needed for rake tasks to execute with your gems.
    # Add your own tasks in files placed in lib/tasks ending in .rake,
    # for example lib/tasks/switchtower.rake, and they will automatically be available to Rake.
    
    require '~/rtenv.rb'
    
    require(File.join(File.dirname(__FILE__), 'config', 'boot'))
    
    require 'rake'
    require 'rake/testtask'
    require 'rake/rdoctask'
    
    require 'tasks/rails'
    
  6. Require your environment file in your Rails’ environment. This will make sure that your gems will be used when starting and Rails command (including the server itself).If you use ~ while requiring your environment file, you will want to also manually set the HOME environment variable before, as sometimes it is not set when the Mongrel service is started from the cPanel.This is also the place to set the relative path of your Rails’ root URL (only if your Rails’ root is in a sub path). For example, my server’s root URL is http://mysite.com/redmine.
    # Be sure to restart your web server when you modify this file.
    
    # Set the relative path to the root URL
    ENV['RAILS_RELATIVE_URL_ROOT'] = '/redmine'
    # Force the HOME environment variable (used for ~ requirements)
    ENV['HOME'] = '/home/username'
    # Require our gem environment
    require '~/rtenv.rb'
    
    # Uncomment below to force Rails into production mode when
    # you don't control web/app server and can't set it the proper way
    # ENV['RAILS_ENV'] ||= 'production'
    
    ...
    
  7. If the cPanel creation of your Rails application succeeded on steps 2 or 3, and if you want to use the default Mongrel version, you should be ready to go. After creating your database, migrating it, etc… you should be able to run your server using cPanel.Don’t forget to create a Rewrite rule using your cPanel if you want it to be accessible without specifying the port.
  8. If you want to use your custom Mongrel version, or if the cPanel creation failed, no worries, everything can be done by hand:
    1. Create a script that will launch your Mongrel server. Don’t forget to add execution permissions to it. This is very easy, for example this is mine:
      #!/bin/sh
      
      cd ~/rails_apps/redmine
      /usr/bin/ruby /usr/bin/mongrel_rails start -p 12006 -d -e production -P log/mongrel.pid
      

      If you want to use your own Mongrel version, you should first setup the GEM_PATH environment variable, and then change the mongrel_rails path to use your own one.

    2. Create a script that will stop your Mongrel server. Don’t forget to add execution permissions to it.
      #!/bin/sh
      
      cd ~/rails_apps/redmine
      /usr/bin/ruby /usr/bin/mongrel_rails stop -p log/mongrel.pid
      
    3. Now you can start and stop your server from your terminal. It will be daemonized, so you can kill your terminal afterwards, your server will still be up.It will be listening on the port you will have specified in the start script (12006 in my example). So you can access it through http://mysite.com:12006/.You can then use those scripts in other scripts, crontabs or any other way you want to start and stop your server.
    4. If you want to use normal URL without specifying the port, and eventually a relative path for your root URL, you will have to create manually the Apache Rewrite rules.This is done in your root .htaccess file:
      RewriteEngine on
      
      RewriteCond %{HTTP_HOST} ^mysite\.com$ [OR]
      RewriteCond %{HTTP_HOST} ^www\.mysite\.com$
      RewriteRule ^redmine/(.*) "http\:\/\/127\.0\.0\.1\:12006\/$1" [P,L]
      
      

      If you don’t want the relative root URL redirection, just remove the redmine/ part from my example:

      RewriteEngine on
      
      RewriteCond %{HTTP_HOST} ^mysite\.com$ [OR]
      RewriteCond %{HTTP_HOST} ^www\.mysite\.com$
      RewriteRule ^(.*)$ "http\:\/\/127\.0\.0\.1\:12006\/$1" [P,L]
      
      

And now you should be set up, with your Rails server running the way you want:

  • with your own Rails version,
  • with your own gems,
  • with your own Mongrel version (if you don’t use cPanel),
  • with the root URL you want for it,
  • with the complete control on when to start and stop it, even without cPanel.

Cheers!

Muriel

Howto, Web development , , , , , , , , , ,

How to make Apache route all requests except some directories ?

Hi

Some context might help:

  • You have an Apache running environment, on which you are not root (therefore no access to the global Apache config). Typically the kind of hosted shared environment. You have full control on the files stored in your www/ directory.
  • You want to root all requests by default to a given URL (might be a directory, another port…).
  • But you also want some directories like www/realdir to be visible in a flat way

So first step, you define a nice www/.htaccess file in your directory to redirect everything by default:

RewriteEngine on
# Redirect every request to the port 12004, hosting a Ruby on Rails server
RewriteCond %{HTTP_HOST} ^mysite\.com$ [OR]
RewriteCond %{HTTP_HOST} ^www\.mysite\.com$
RewriteRule ^(.*)$ "http\:\/\/127\.0\.0\.1\:12004\/$1" [P,L]

Then all your http://mysite.com/my/path/ URL will be routed to your alternate location (a nice Ruby on Rails server running on port 12004 for this example).

But then you want to get your www/realdir/index.html file to be accessible using the normal http://mysite.com/realdir URL. By default, Apache will also route this URL to your alternate location. And you want Apache to do it, not your alternate location.

The trick is to declare a simple .htaccess file in your directory, declaring the Rewrite engine.

RewriteEngine on

And that’s it, you can use this way on any directory you want !

Muriel

Howto, Web development , , , , , , ,

Test Title: The Big Jump

So here it is: my first post on this blog. Usually it comes down to “Test Title” and “Test Content“. I will try to make it more interesting by introducing the purpose of this blog and giving some details about my past experience.

I am a developer who decided to make the big jump out of the common easy-salary-and-blind-job-company-sponsored way of life, to the more woah!-I-am-my-boss-but-no-salary-self-sponsored one.

I had several motivations that made me resign from my daily work at BigCorporationBigBucks(TM) (around 5000 employees worldwide):

  1. First, I had more and more the bitter feeling that whatever the efforts developers could make, the company would always have the same profile. As a developer, I have always wanted to reach big achievements (I am a bit of this idealistic kind), and I observed that when I was reaching those achievements… well there was not so much difference on the company’s success or failure. That was frustrating.
  2. Second, I realized there is a strange way of thinking in many companies handling with software development: “Developer” is not seen as a simple competency there. It is also associated to the idea of “newbie”. There is a kind of common thinking that a developer starts his career as a “newbie developer”, then after a few years of successful experience he will become a “manager” then “director”, and the list goes on depending on the pyramidal structure of the company.

    Other jobs are a little different: a “newbie architect” will become a “skilled architect”, a “newbie pianist” will become a “skilled pianist”, and their reputation and recognition will follow according to that.

    In BigCorporationBigBucks(TM), I have met some highly skilled developers, and they had basically 2 choices at a given point of their career:

    • First case:


       

      I have seen this scenario several times, and it always ends up the same: either the developer ruins a whole department because he has no management skills, or after a few months he will ask to get his developer position back because he doesn’t like to fill in Excel spreadsheets and argue over political arguments all day long: he just wants to develop projects.

    • Second case:




      Imagine the same scenario in another context: “Hey little Yehudi, you really are gifted with playing the violin. Therefore we can offer you a prime time at our big music TV channel where you will dance hip-hop. Leave your violin at home.“.

      Well, this is what BigCorp is doing every day (and this can also explain some hip-hop dancers’ performance on BigCorpTV(TM) ;-) )

    Manager is a real hard working job, with all its theories, experiences, surprises, specific competences… so is developer. You can’t replace one with another unless you perform real hard. A manager with no development skills will ruin a software, and a developer with no management skills will ruin a team’s work. Seems simple, but I did not want to work anymore in an environment that did not respect neither developers nor managers.

  3. Third, I began more and more inclined towards Open Source development. I was more and more convinced that a company’s added value does not rely in keeping its source code secret.A company producing software is doing so for a customer market. Sooner rather than later, other companies will also target this market. Among those other ones, you will find Open Source software. If your added value is not present in additional services, your customers will always find a similar software for free.Closed source will then forcefully come to a decline and you will have worked very hard with your team of 10 people on a project that can’t compete with its equivalent Open Source gathering 100 people. Therefore I began to loose faith in companies whose declared added value lied in its source code exclusivity.
  4. Fourth, as a little idealistic as I am, I was kind of seduced by the Open Source philosophy. People developing software for everybody to be seen, tested, used, improved is grand. This is called giving the power to people.
  5. Last, but not least, I had the opportunity (chance?) to make the big jump. Save some cash, then go.

And so here am I, 3 years later. I have contributed to many Open Source projects, I have created a dozen, I have taken enough time to grow my competences into various fields, new programming languages, new technologies, marketing, finances… And I still can’t draw better than a 5 year old child 😉

I am now a freelance developer, and even if cash does not flow as if I was in BigCorp, I have still kept my initial vision of development and strive to live from it.

In this blog, I will talk about several things:

  • Technical challenges I encounter, as it can be useful for others and will also be a way for me to not hit my head twice on the same wall.
  • Various thoughts on development, business, projects.
  • My own experience, as I hope it can be useful for others.

Too long already for a first post. I will have to learn how to be more concise ;-)

Cheers

Muriel

My life , , , , , , , , ,