Cookie Based API login using Guzzle

We've all used curl to consume external services - scraping, APIs, etc. We know that curl is awesome, but there is that feeling that you should be doing this in a more abstract way, or using something not too hardcore.

Yesterday, I had to connect to an external API using cookie based authentication. I initially thought of doing it with curl, but I remembered Guzzle, and wanted to give it a try.

Hello Guzzle

Guzzle is a PHP HTTP client that makes it easy to send HTTP requests and trivial to integrate with web services. 

In pseudocode:

• create a guzzle client
• create a cookie 
• login and update that cookie
• use that cookie for next requests

Here is an example:

1:  // Create a Client   
2:  $client = new \GuzzleHttp\Client\Client([  
3:    'base_url' => 'http://someserver.com',  
4:    'defaults' => [  
5:      'headers' => [  
6:        'Accept' => 'application/json'  
7:      ]  
8:    ]  
9:  ]);  
10:    
11:  // Create a new cookie  
12:  $cookieJar = new \GuzzleHttp\Cookie\CookieJar();  
13:    
14:  // Send a post  
15:  $request = $client->createRequest('POST', '/api/login', [  
16:    'cookies' => $cookieJar,  
17:    'body' => [  
18:      'UserName' => 'USERNAME',  
19:      'Password' => 'PASSWORD',  
20:    
21:    ]  
22:  ]);  
23:    
24:  // ....  
25:    
26:  // Reuse the cookie for next requests  
27:  $response = $client->get('/api/user', [  
28:    'cookies' => $cookieJar  
29:  ]);  

You can store the cookie in a shared session so that you don't need to login each time you do a request.

Links:

• Fetching a URL with Cookies using curl: http://docstore.mik.ua/orelly/webprog/pcook/ch11_04.htm

Apache reverse proxy with ProxyPass how to

First install and enable Apache reverse proxy module (mod_proxy) using apt-get (ubuntu, debian, etc):

~> sudo apt-get install libapache2-mod-proxy-html
~> a2enmod proxy_http

Edit your virtual host:

Configure the module to allow connections from your network, edit /etc/apache2/mods-enabled/proxy.conf

Restart apache and you are ready to go!.

Possible errors you could get

If you get this error in /var/log/apache2/error.log :

[error] [client 192.168.56.1] client denied by server configuration: proxy:http://127.0.0.1:28017/folder/

You have to check the "Allow" directive in /etc/apache2/mods-enabled/proxy.conf  remember to allow your network!!!.

If you get this error in /var/log/apache2/error.log :

[warn] proxy: No protocol handler was valid for the URL /api/. If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.

you should remember to enable the module:

~> a2enmod proxy_http

How to start working with ZfcUser (ZF2)

I will explain how to setup a Zend Framework Application from the skeleton app in github and then how to install and configure the module called ZfcUser .

The purpose of ZfcUser is:

The purpose of ZfcUser is to provide a flexible foundation for the common practice of user authentication / identification and registration. Ultimately, a user will authenticate via some authentication adapter, and upon successful authentication will be identified by a populated user model for the given session. The focus of ZfcUser is flexibility and more importantly, extendibility. ZfcUser should be able to successfully serve as a starting point for authentication in a very large percentage of ZF2 web applications with widely varying names.

So lets start...

Requirements for this howto:

• PHP 5.3.3+ (with pdo-sqlite or pdo-mysql), 
• I'm using php 5.4 that has a built in webserver (note: use it only for dev purposes!!!). 
• Git
• sqlite

Install Zend Framework 2 Application 

The process is described here, but basically you install composer:

~> curl -s https://getcomposer.org/installer | php --

Then install the application using composer:

~> php composer.phar create-project --repository-url="http://packages.zendframework.com" zendframework/skeleton-application

There are alternate methods in the skeleton application github page.

At this point we will have ZF2 MVC application installed and working, can check this using the php built in webserver (5.4+),

~>php -S localhost:8888 -t public/

and browsing to http://localhost:8888 with your browser:

Install ZfcUser

Edit the composer.json file and add this line to require ZfcUser. Composer will automatically install ZfcBase thats a dependency of ZfcUser.

composer.json should look like this:


{
    "name": "zendframework/skeleton-application",
    "description": "Skeleton Application for ZF2",
    "license": "BSD-3-Clause",
    "keywords": [
        "framework",
        "zf2"
    ],
    "homepage": "http://framework.zend.com/",
    "require": {
        "php": ">=5.3.3",
        "zendframework/zendframework": "2.*",
        "zf-commons/zfc-user": "dev-master"
    }
}

Then update the dependencies:

~> php composer.phar update

Configure Zend Framework to work with ZfcUser

Add those modules (ZfcBase and ZfcUser) to the application configuration so they can be bootstraped and configured, editing /config/application.config.php:

As you can see in your browser the module is active, browse http://localhost:8888/user

If you try to Sign up, clicking in the link you will receive an Application error, and that simply because we didn't configure the database yet.

Let's setup the database with the schema that comes with ZfcUser. You can locate the folder /vendor/zf-commons/zfc-user/data . ZfcUser provides 3 schemas, one for postgres, one for mysql and one for sqlite. We will use sqlite configuration: schema.sqlite.sql .
From the root folder of the application, run:

~> cat vendor/zf-commons/zfc-user/data/schema.sqlite.sql | sqlite3 data/users.db

Now we need to configure the database adapter so the module can recognize it.
Create a file called database.local.php in the /config/autoload folder and add the configuration:


<?php
return array(
    'service_manager' => array(
            'factories' => array(
                'Zend\Db\Adapter\Adapter' => 'Zend\Db\Adapter\AdapterServiceFactory',
            ),
    ),
    'db' => array(
        'driver' => 'pdo',
        'dsn'    => 'sqlite:'.getcwd().'/data/users.db',
    ),

);

And bam! its done, i can signup:

And it even get my avatar from gravatar.com !!!

Fantastic!!!

Check the wiki to override the default behavior:

https://github.com/ZF-Commons/ZfcUser/wiki

Share a console in linux with multiple users using screen

Sometimes you need to see or share some actions that you or somebody is doing in a remote server terminal (lx, mac..).
You can share your screen (join.me is a good solution) but the real way of doing it is to share you terminal / console.

The best way of doing this is using screen  multiuser option.
The procedure es staightforward:

1. open a screen session (-S give a session name):

screen -S screen-multi

2. activate multiuser mode

CTRL-A

 :multiuser on

3. the other user (connected to the same server) has to:

screen -x screen-multi

And thats it, both will see the same session and can write at the same time on it.

More detail about this procedure: http://ubuntuforums.org/showthread.php?t=299286

Show ping summary in MacOS terminal

ping can be used for a lot of things. I do use it a lot to test reachability and how my internet access is behaving.

Usually you just execute ping in your terminal and then hit Ctrl+C to see the summary as follows:

Hitting Ctrl+C will send an interrupt signal (SIGINT) to the process, making it to terminate and show ing the summary.

Once in a while you need to check a long ping run but don't want to stop it (because of the final stats). The way of doing it, is to send a SIGINFO signal hitting Ctrl+T while the process runs.

Check line 9 and 10 in the previous example:
and ping continues running!!!

References: Ping OS X Manual.

Temporarily ignoring files

There is a lot of times that you want to ignore a file that was already committed in your git repo:

git update-index --assume-unchanged <file>

when you want to rollback this and start tracking changes again:

git update-index --no-assume-unchanged <file>

Read more about this: http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html

Breadcrumbs with Zend_Navigation

I needed to create a breadcrumb into an application im developing, and i thought that would be cool to do it with Zend Framework and not just reinvent the wheel.

There is Breadcrumb helper that we can use, its very easy to setup, so here we go:

Setup navigation.xml

First we need to create an xml with our navigation map. In this case i want to recreate something like this:

• Home - uri: /
• Search - uri: /search
• Management - uri: /management
• Parameters - uri: /management/parameters
• Parameter Detail - uri: /management/parameters/view/XYZ

The idea is to create a breadcrumb like this:

[ Home ]
[ Home / Search ]
[ Home / Management ]
[ Home / Management  / Detail ]

So i will create a /application/configs/navigation.xml as follows:

As you can see it represents the navigation map i want in the breadcrumb. Each uri specifies an option in our navigation map.

Bootstrap

You have to bootstrap the navigation, here is the init method for your Boostrap.php :
Check on line 12 the 'nav' parameter, its the first element that correspond to the configuration section we want to use.

Add it to your layout

After setting up our navigation, we need to add the breadcrumb to our layout template. My layout file looks like this:



This is the basic configuration im using:

• setMinDepth(0) gets and sets the minimum depth a page must have to be included by the helper. Setting NULL means no minimum depth.
• setLinkLast(false) I don't want a link in the last option of the breadcrumb.
• setSeparator(" / ") sets text with which to separate aggregated content.

Select the options in our controllers

Finally we need to tell Zend_Navigation where we are so it can show the right breadcrumb options.

To activate the breadcrumb on every action of a controller we need to get the uri and then pass it to the navigation object:

If we are in some weird uri and want to show /search in the breadcrumb you just have to use the uri parameter to search the right option.

This is the final result (using Twitter Bootstrap styling):

Hope it helps.

Mac OS X could not be installed on your computer

Today I upgraded my MacBook Pro operating system to OS X Lion, and was not so "cool". I get this error in the first attempt:

Install Failed

Mac OS X could not be installed on your computer

Mac OS X can't be installed on the disk Macintosh HD, because a
recovery system can't be created. 
Visit www.apple.com/support/no-recovery to learn more.

Going to www.apple.com/support/no-recovery is useless, will not help you at all (WTF Apple!!!)

To solve this issue you just have to:

1. Open Disk Utility in Application -> Utilities -> Disk Utility
2. Select your disk, in my case its: Machintosh HD
3. Click in "Repair Disk Permissions" and wait till it finishes
4. Start the upgrade again

Hope it helps!

Throw a 404 in Zend Framework

This is very easy, just throw an exception in the controller action:


This will generate a 404 page using the ErrorController.

How to merge selected files from a branch in git

I've being learning git for some time and i love it, i will start blogging some tips and tricks about it.

First you need to bookmark this: http://gitref.org/

Now, how to merge selected files from a branch to my master branch?
Move to the branch you want to "merge to", and:

Zend Framework: How to access your configuration from the application.ini file

There are many ways of getting the config from our application.ini file. The usual way is to load it using Zend_Config in our Boostrap class. The other way is to load it from a parameter in the front controller (Zend_Application put it there).

Using Bootstrap
Add this to your Boostra.php file:


You can fetch the config wherever you want using: 

$config = Zend_Registry::get("config");

Loading from Front controller
From a View action:

From outside the controller:

Sphinx Search - why and how to use sphinx search delta indexes

Problem:
Anybody who used sphinx search knows that reindexing big indexes takes a long time. The main problem here is that the whole index is recreated every time you execute a reindexing.

Solution:
The way of handling this is by using delta index updates with index merging. The idea is to have 2 indexes:

• the "main" and big index for the old (unchanged) data, and 
• a small "delta" for the new (recently changed) data.  So, instead of reindexing the "main" index, you should only reindex the "delta" one every few minutes.

After a while (once a day) you should merge the "delta" and the "main" index (depending of the size of the delta).
This is called "main+delta" scheme.

How-to:
Having a table "documents" with fields: "id, title, body",  we create the sphinx search index as follows:


As you can see the main sql_query give us all the documents from the documents table. The idea of the "main+delta" scheme is that every time you reindex main, you will have to store the last id processed somewhere, so delta can start from there and process a little amount of records.

So first,  create this table in mysql to store that id:

Then in sphinx.conf:

How does this work? As you can see in main specification a pre-fetch query called sql_query_pre appears. That query is executed before the main query (you can have a bunch of those).

REPLACE INTO sph_counter SELECT 1, MAX(id) FROM documents

Our pre-fetch query, updates a record in the sph_counter table to be used later by delta.  This record will store the max(id) from our documents table in the moment of indexing, so the main query will get documents with id less than or equal to that maximum and delta query with ids bigger than that max.

In brief, you update main once in a while, and delta every so often. main will get ALL the documents till the time you update it, and delta will get all the new ones.

In your code you will have to search in both indexes:

$sphinxClient->Query(“this is my search query”, “main delta”);

Indexing main will take a long time too, but you will be sort of "live" because delta will update very quickly.
Instead of reindexing main you can merge both indexes, and only update delta. I will write about merging in a future post.

Links:

• http://sphinxsearch.com/
• http://www.ivinco.com/

How to reset MySql root password on ubuntu

Just do this

sudo dpkg-reconfigure mysql-server-5.1

Works on debian too.

How to create a .iso image of a cd on a Mac OsX

As OsX is Unix based we have a lot of tools that came in the standard installation of our Mac.
To create an ISO file we have to use the dd command in our terminal.

Insert your CD-ROM into your CD-ROM drive, and then fire up your Teminal (Applications/Utilites/Terminal). Run this command to get the name of the device: drutil status

diegoMB:~ diego$ drutil status
 Vendor   Product           Rev 
 HL-DT-ST DVDRW GWA4080MA   BE39

           Type: DVD-ROM              Name: /dev/disk1
       Sessions: 1                  Tracks: 1 
   Overwritable:   00:00:00         blocks:        0 /   0.00MB /   0.00MiB
     Space Free:   00:00:00         blocks:        0 /   0.00MB /   0.00MiB
     Space Used:  336:26:02         blocks:  1513952 /   3.10GB /   2.89GiB
    Writability: 
      Book Type: DVD-ROM (v1)

As we can see here our drives name is disk1. To use dd we need to umount the CD-ROM:

diskutil unmountDisk disk1

Now we can use dd to create the iso file:

dd if=/dev/disk1 of=my_file.iso

after the iso file is created you can mount the CD-ROM with:

diskutil mountDisk disk1

;)

Tips and Tricks - MYSQL - how to dump a database

So you want to backup your mysql database??? Mysql has many ways of doing it, i will explain the easier one: mysqldump.


Being strict, msqldump is not a "backup utility", it creates a script with  "CREATE TABLE" and "INSERT INTO" commands that you can run in your server to re-create your database.  mysqldump  is shipped with MYSQL so everybody should have it installed.

Using mysqldump command line is very straightforward, just type this in your command line or terminal (using your username and password):

mysqldump -u USERNAME -pPASSWORD OUR_DATABASE > filename.sql

This will give us a file (filename.sql) containing "CREATE TABLE" and "INSERT INTO" commands for OUR_DATABASE.
Some times you need to dump ALL your databases (you are moving you dev machine) in your MYSQL server:

mysqldump -u USERNAME -pPASSWORD --all-databases > filename.sql

If you are using MAMP (in macOS), your mysqldump binary is located in /Applications/MAMP/Library/bin/mysqldump.


There is a lot more options here in the official MYSQL documentation: http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html


To restore our database, we have to we have to run this script in our MYSQL server. You can do it using any utility (phpmyadmin for instance) or jus (again) the command line:

mysql -u USERNAME -pPASSWORD DATABASE_NAME < filename.sql

So, good backup!! ;-)

  

Crear un demonio con PHP y PEAR

 Hoy tuve la necesidad de crear un demonio para WHObyYOU por lo que voy a compartir con uds el resultado.
Un demonio es un proceso / programa que corre en segundo plano (de forma pasiva). La mayoría de la veces, cuando uno necesita correr un proceso continuamente; termina haciendo tareas programadas (cron jobs) con las ventajas y desventajas de este tipo de implementación. La mayor desventaja de una tarea programada es el solapamiento de la tarea con otra. Hay que hacer alguna magia para que esto no pase.
Con demonios esto no importa ya que una tarea no comienza hasta que empieza la otra (dentro del mismo proceso).

El ejemplo que voy a mostrar ahora es simplemente para mostrarles como hacer el demonio con PEAR::System_Daemon.

Primero hay que instalar pear, en Unbuntu

sudo apt-get install php-pear

Luego instalar PEAR::System_Daemon

sudo pear install -f System_Daemon

Luego de instalar el paquete, lo unico que tenemos que hacer es :

require_once "System/Daemon.php"; // Include the Class

System_Daemon::setOption("appName", "mydaemon");

System_Daemon::start();

A continuación muestro un script de ejemplo, que lo unico que hace es grabar en un log cada 5 segundos con un máximo de 5 veces.



Paso a explicar las diferentes partes del script:

En la línea 1 indicamos que este script se ejecutará con PHP.
En la linea 3 incluimos la clase System_Daemon de par.

En la linea 5 configuramos el demonio, "appName" y "appDir" son los únicos requeridos para que funcione.

$options = array(

    "appName" => "pruebaDiego",

    "appDir" => dirname(__FILE__)

);

Lineas 13 y 14 creo el demonio (crea un proceso hijo, se separa del proceso padre y el proceso padre muere).

Luego creo un while(true) y le genero una condición de salida: podría ser por una falla del programa, captura de una excepción, cualquier cosa. En este caso la condición de salida se da cuando el bucle se ejecutó 5 veces.

En la línea 21 escribo en el log. El System_Daemon nos ofrece una forma simple de escribir un log, en este caso el archivo de log es:

/var/log/pruebaDiego.log

Para ejecutar el script simplemente, hay que ejecutar

$ sudo ./test.php 

Luego podemos ver el log:

$ tail -f /var/log/pruebaDiego.log

Por más información:

• http://pear.php.net/package/System_Daemon/docs
• http://kevin.vanzonneveld.net/techblog/article/create_daemons_in_php/ 

  

login ssh sin clave

Es muy común preguntarse como hacer para conectarnos de forma automática a equipos remotos para por ejemplo correr procesos.
O simplemente cómo hacer para entrar a otro equipo desde el nuestro sin tener que poner la clave continuamente. Normalmente uno hace lo siguiente para hacer esta conexión:

usuarioA@diego:~$ ssh usuarioB@servidor

usuarioB@servidor's password:  

Luego ingresamos la clave de usuarioB.  Si tenemos que hacer esto todo el tiempo llega un momento que es bastante tedioso, y peor si tenemos que conectarnos a muchos equipos todos con diferentes claves.

Para solucionar esto, primero tenemos que crear un par de claves en nuestro equipo (o desde donde nos queremos conectar) de la siguiente forma (mi sistema operativo es Ubuntu 10.10):

usuarioA@diego:~> ssh-keygen -t rsa

Si no queremos que nos pida una clave cuando nos conectemos simplemente presionar Enter (esto es muy útil para hacer scripts luego). Esto genera una clave pública y una privada en el directorio ~/.ssh

Luego tenemos que copiar la clave publica al equipo al que queremos conectarnos, en este caso el equipo se llama "servidor" con el usuario "usuarioB". Primero tenemos que crear la carpeta ~/.ssh en servidor:

usuarioA@diego:~> ssh usuarioB@servidor mkdir -p .ssh 

usuarioB@servidor's password: 

Luego hay que agregar nuestra clave publica al  .ssh/authorized_keys de servidor :

usuarioA@diego:~> cat .ssh/id_rsa.pub | ssh usuarioB@servidor 'cat >> .ssh/authorized_keys'
usuarioB@servidor's password: 

Ahora nos podemos conectar directamente sin ingresar la clave

usuarioA@diego:~$ ssh usuarioB@servidor

usuarioB@servidor:~$

Introducción a Zend_Queue

Esta semana tuve que armar un Queue para un proyecto en el que estamos trabajando (WHObyYOU.com) y obviamente intenté hacerlo con Zend_Queue.

En este caso necesitaba consumir en "tiempo real" la Streaming Api de Twitter (ya haré un post sobre esto),  por lo que necesitaba procesar los tweets fuera el proceso que consumía el stream.

Zend_Queue es bastante simple y no tiene mucho misterio. En este caso vamos a usar el adaptador de Base de Datos (Zend_Queue_Adapter_Db) para guardar nuestros mensajes.

Primero creamos las tablas que necesitamos (queue y message), en el caso de mysql:


La defincion de tablas la pueden encontrar en Zend/Queue/Adapter/Db/mysq.sql

Creamos el queue utilizando una clase abstracta:



Luego agreguemos un Tweet al queue, supongamos que un Tweet es una array:



Luego a procesar los mensajes. En el caso de este proyecto, hicimos una tarea programada (cron) que corre cada 30 segundos y procesa el queue. Zend_Queue retorna los mensajes usando Zend_Queue_Message_Iterator (lo pueden ver en la documentacion).



Esto se puede mejorar mucho creando nuestros propios adaptadores, eso vendrá en una próxima entrega.