Packt is celebrating #eBookBonanza! All eBooks and Videos are available for just $5! VISIT http://bit.ly/1jdCr2W NOW! #Packt$5
Book Review: Using OpenRefine
[The publisher of this book provide me with a free e-copy for review.]
Link to the book: http://www.packtpub.com/openrefine-guide-for-data-analysis-and-linking-dataset-to-the-web/book
My familiarization with OpenRefine started with Google Refine. The main idea has evolved and become open for contributions of the community.
This book is a nice cookbook, easy to follow written in a friendly language, to perform both simple and complex operations over semistructured data such as HTML tables, spreadsheets, csv files, among others, exploiting the linkeability of your datasets with the Linked Open Data (LOD) cloud. Even without previous knowledge any user can take this book and from scratch start to use OpenRefine.
Organization:
The book is divided into four chapters plus an appendix.
Chapter 1: Presents the first set of easy-to-follow recipes to get your hands into loading and preparing your data.
Chapter 2: The main contribution is on how to sort and create facets to select (or isolate) data based on regular expressions over the cell values, with the main goal of fix the datasets.
Chapter 3: Tells you how to deal with more advanced operations over your data, and gives a brief introduction to GREL, the language defined to manipulate cell values. GREL is simple and powerful enough to match and replace the cell’s content as needed by different purposes, that’s because they have an appendix with a deeper explanation.
Chapter 4: Once that all the data have been normalized this can be reconciliated with an external knowledge base or linked open dataset such as Freebase (can be whatever knowledge base appropriate for the data at hands) to enrich the semantics of the data.
What I like about this book:
The format of cookbook and the easy reading are the most positive aspects of this book. The authors made a good work explaining all the use cases using a real data example.
What I dislike about this book:
The smaller things that I dislike are part of the format used in the book. It would be better to have enumerated figures and sections to make references effortless. However, the figures are very self explicatives and ad-hoc with the explanations. Also, for easy referencing would be nice to have a consecutive recipes numeration and not a new numeration in each chapter.
Wrapping up, I would strongly recommend this book to any user interested in: Semantic Web/Linked Data, Linked Open Data publication, Data Integration, Named Entity Recognition; at student, lecturer, practitioner level, or just for hobby.
Reducing space Theorems, Proofs, Lemmas, etc in LaTex
If you need to make more space in your papers (space is valuable in this cases), because the (vertical) space between theorems, proofs, lemmas, remarks, examples, and definitions with other paragraphs is too large. Here I will show a little trick to do that.
We need to import the asmthm package. But, not too fast, this will trigger an error. So, above the import we need to add two lines that solve the bug related with an incompatibility in the proof enviroment (or macro).
Then we are ready to define our theorem style with the desire spaces and fonts as follows:
\let\proof\relax
\let\endproof\relax
\usepackage{amsthm} %http://ctan.org/pkg/amsthm
\newtheorem{theorem}{Theorem}
\newtheoremstyle{exampstyle}
{\topsep} % Space above
{\topsep} % Space below
{} % Body font
{} % Indent amount
{\bfseries} % Theorem head font
{.} % Punctuation after theorem head
{.5em} % Space after theorem head
{} % Theorem head spec (can be left empty, meaning `normal')
\theoremstyle{exampstyle} \newtheorem{example}{Example}
\theoremstyle{exampstyle} \newtheorem{remark}{Remark}
\theoremstyle{exampstyle} \newtheorem{definition}{Definition}
\theoremstyle{exampstyle} \newtheorem{lemma}{Lemma}
This code will enumerate each enviroment –theorem, proof, etc– independly. If you are wishing to enumerate relatively to sections you have to use these modifications.
\let\proof\relax
\let\endproof\relax
\usepackage{amsthm} %http://ctan.org/pkg/amsthm
\newtheorem{theorem}{Theorem}[section]
\newtheoremstyle{exampstyle}
{\topsep} % Space above
{\topsep} % Space below
{} % Body font
{} % Indent amount
{\bfseries} % Theorem head font
{.} % Punctuation after theorem head
{.5em} % Space after theorem head
{} % Theorem head spec (can be left empty, meaning `normal')
\theoremstyle{exampstyle} \newtheorem{example}[theorem]{Example}
\theoremstyle{exampstyle} \newtheorem{remark}[theorem]{Remark}
\theoremstyle{exampstyle} \newtheorem{definition}[theorem]{Definition}
\theoremstyle{exampstyle} \newtheorem{lemma}[theorem]{Lemma}
In the style you can customize the environments with your preferences and even create more than one style if necessary.
Cheers
Using Algorithm2e in ACM template
If you have tried to use this package to create algorithms in the ACM template for papers, you must be come across this issue. If you only add the packate algorithm2e with something like
\usepackage[ruled,vlined]{algorithm2e}
you should have received a nice errors telling you “1616:Too many }’s” and “1617:Extra \fi \fi”.
To solve this annoying bug, that could make you lose a lot of valuable time, just add the following code before the import of the package.
% For algorithms in ACM template
\makeatletter
\newif\if@restonecol
\makeatother
\let\algorithm\relax
\let\endalgorithm\relax
\usepackage[ruled,vlined]{algorithm2e}
I hope this will be useful for you like was for me 🙂
Cheers
Sorting HashMap based on its values
This task could be done in different ways. Here a discussion about some ways.
In this post I will show one that I followed. Roughly speaking, the idea is convert the map into a list with the purpose of use the classic Collection.sort() method.
The first think is that you need is to define a new class, that represents one map entry; and implements the Comparable interface. In this class you have to implement the compareTo method and override the toString (if you want).
import java.util.Map;
@SuppressWarnings("rawtypes")
public class CustomEntry implements Comparable
{
private Map.Entry entry;
public CustomEntry(Map.Entry entry)
{
this.entry = entry;
}
public Map.Entry getEntry()
{
return this.entry;
}
public int compareTo(CustomEntry anotherEntry)
{
Integer thisIntegerVal = (Integer) (this.getEntry().getValue());
int thisVal = thisIntegerVal.intValue();
Integer anotherIntegerVal = (Integer) (anotherEntry.getEntry().getValue());
int anotherVal = anotherIntegerVal.intValue();
return (thisVal < anotherVal ? 1 : (thisVal == anotherVal ? 0 : -1));
}
public int compareTo(Object o)
{
return compareTo((CustomEntry) o);
}
@Override
public String toString()
{
StringBuilder str = new StringBuilder();
str.append(this.getEntry().getKey()).append(":").append(this.getEntry().getValue());
return str.toString();
}
}
Next step is to define the function that will convert any map into a list. Basically, this function will recover the entries in the map, and transform them into custom entries using the previous class CustomEntry.
import java.util.*;
import java.util.Map.Entry;
public class MapFunctions
{
public MapFunctions()
{
}
public static <T, E> List<CustomEntry> convertMapToList(Map<T, E> map)
{
List<CustomEntry> list = new ArrayList<CustomEntry>();
Set<Entry<T, E>> entrySet = map.entrySet();
Iterator<Entry<T, E>> iterator = entrySet.iterator();
while (iterator.hasNext())
{
Map.Entry<T, E> entry = (Map.Entry<T, E>) iterator.next();
CustomEntry customEntry = new CustomEntry(entry);
list.add(customEntry);
}
return list;
}
}
Now, you are allowed to execute the following instructions:
THashMap<String, Integer> map = ... /* some initialization */ List<CustomEntry> sortedList = MapFunctions.convertMapToList(map); Collections.sort(sortedList); System.out.println(sortedList);
and you will print the list of entries (from your original map) sorted based on its (original) values.
Cheers.
How to recover picture files deleted from your SD Card
Recently some of my photos were inaccessible in the SD Card […sad face…] Without options I formatted the SD Card, giving me up and resigned to losing my photos. When I mentioned this situation today at dessert time someone mentioned the file recovers … and I said why don’t try the last time? Looking for some file recovery for Linux (of course) I found the magic “PhotoRec, Digital Picture and File Recovery“. It can be installed directly from the console:
sudo apt-get install testdisk
Insert your SC Card or connect your camera to the PC and execute photorec by typping
sudo photorec
Then you will see an screen with all the SD Card devices then you have to select wish to recover.
Another screen to select the specific partition, the types of file you wish to be recover and where do you wish to leave the files recovered. After that PhotoRec start to do its magic.
Finally, I could recover my photos and I say thanks PhotoRec 🙂
This is the wikipage of the open source PhotoRec and another post orienting on how to use this application.
An important feature is that you can recover any type of files.
Best.
Eclipse problem with java-6-openjdk-amd64
Hi all,
with the last upgrade of java-6-openjdk-amd64 some problems have raised.
In this links you can see some of the bug reports:
http://askubuntu.com/questions/186986/eclipse-has-multiple-issues-after-jre-6-openjdk-upgrade
http://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg3807958.html
The error is triggered each time that you want to use the JRE or start a server.
An internal error occurred during: "Launching application-name". org/eclipse/jdt/debug/core/JDIDebugModel
There are two possible quick solutions:
-
sudo apt-get --reinstall install tzdata-java
- Put this line in eclipse.ini:
-Dcom.ibm.icu.util.TimeZone.DefaultTimeZoneType=ICU
This will resolve the issue.
Cheers
Path of Ubuntu Applications
Usually I’m looking for the path where ubuntu installs some applications, here is a way to do it.
$ sudo dpkg -l | grep name_of_the_app
For example if you want to find what installed applications contains the word ’emacs’ you have to use this
$ sudo dpkg -l | grep emacs ii emacs 23.3+1-1ubuntu9 The GNU Emacs editor (metapackage) ii emacs23 23.3+1-1ubuntu9 The GNU Emacs editor (with GTK+ user interface) ii emacs23-bin-common 23.3+1-1ubuntu9 The GNU Emacs editor's shared, architecture dependent files ii emacs23-common 23.3+1-1ubuntu9 The GNU Emacs editor's shared, architecture independent infrastructure ii emacsen-common 1.4.22ubuntu1 Common facilities for all emacsen ii python-ropemacs 0.6c2-4 Emacs mode for Python refactoring
And then you can find all the files that this application contains.
$ sudo dpkg -L emacs23 /. /usr /usr/share /usr/share/doc /usr/share/doc/emacs23 /usr/share/doc/emacs23/copyright /usr/share/doc/emacs23/README.Debian /usr/share/lintian /usr/share/lintian/overrides /usr/share/lintian/overrides/emacs23 /usr/share/menu /usr/share/menu/emacs23 /usr/share/emacs /usr/share/emacs/23.3 /usr/share/emacs/23.3/etc /usr/share/emacs/23.3/etc/DOC-23.3.1 /usr/share/man /usr/share/man/man1 /usr/share/applications /usr/share/applications/emacs23.desktop /usr/bin /usr/bin/emacs23-x /usr/share/doc/emacs23/changelog.Debian.gz /usr/share/man/man1/emacs23-x.1.gz /usr/share/man/man1/emacs23.1.gz /usr/bin/emacs23
Python and HTML Processing
Trying to harvest some links from the Coursea Lectures, in a fast and efficient way saving me valuable time, I decided to use Python with Beautiful Soup.
Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. It parses anything you give it, and does the tree traversal stuff for you. You can tell it “Find all the links”, or “Find all the links of class externalLink”, or “Find all the links whose urls match “foo.com”, or “Find the table heading that’s got bold text, then give me that text.”
First, you need to download the library from here and untar the bs4 folder in your workspace.
The library is really simple and intuitive to use, since represent the HTML document as a nested data structure.
Too much chatter, now I’m going to show you the python code 🙂
#!/usr/bin/env python
import urllib, sgmllib
# Get a file-like object for the Coursea Web site course page.
f = urllib.urlopen("https://class.coursera.org/gametheory/lecture/preview")
# Read from the object, storing the page's contents in 's'.
s = f.read()
# Here is the import for the library above described.
from bs4 import BeautifulSoup
# BeautifulSoup object, which represents the document as a nested data structure
soup = BeautifulSoup(s)
# Find all the link with rel 'lecture-link'
for videos in soup.find_all('a', rel="lecture-link"):
link = videos.get('href')
# Get the page apointed by the previous link in the same way
f1 = urllib.urlopen(link)
s1 = f1.read()
soup1 = BeautifulSoup(s1)
# Find all the source tags in the document and filter for 'video/mp4'
for link in soup1.find_all('source'):
if link.get('type') == "video/mp4":
# You can concate 'wget' to build immediately an script
print 'wget '+link.get('src')
A full documentation can be found here.
Ensure you have the bs4 folder and a file with the previous code in the same workspace. Then you can execute your python file as usual and get the links to download the video lectures that you want.
Cheers!
MPI debugging
Debug a program is one of the most complicated things when you program. Now if you add a distributed environment as MPI, is horrible. Fortunately, there are many approaches for this issue.
Based on my experience with MPI, the best solution is configured Eclipse with Parallel Tools Plataform (PTP). In this case, you can test your MPI program, deploy them into a cluster and even track the execution (course, not very useful on MPI). Watch here
But, if your time is very short and you don’t want waste it configuring Eclipse, you can use GDB. To connect your MPI program you must use xterm. See below
mpirun -np xterm -e gdb ./program
The only thing you need is configured your hosts computer (where MPI daemon is running and also working nodes) to allow that they accept xterm connection.
Another choice is use valgrind, but only works with openMPI.
Here you can find some tips Click me
Regards
Swap file
Unix system can use special memory section as interchange space called swap memory. By defect, the OS sets swap memory at installing, creating a whole partition with this propose. Course, you can abort this process, letting the system with no swap memory.
For technical reason, is recommendable use swap memory. If the system hasn’t swap memory you can set it by two ways, creating a specific swap partition or a swap file which is file created inside file system, but is used as swap memory.
To create a swap file in linux, the commands are described below:
First you need create a file for swap. Is recommendable create it into root (/) folder
sudo dd if=/dev/zero of=/swapfile bs=1M count=256
Then you must format it
sudo mkswap /swapfile
Finally activate it
sudo swapon /swapfile
You also can add to your fstab file for automatic booting
Ant buildfile
Ant is like “java makefile” to compile .class o jars. It was developed by Apache. It is very useful because you can decide how compile your program just doing few changes in build.xml file.
For my “C” background I love developing libraries and used them into new projects. Unfortunately, java is a little different. There are no “/usr/lib” folder for java jars (maybe, there are, but I don’t know them) so you can’t just put the jars into a common path for all projects, instead you need linking all necessary jars directly.
But, as “C” you can link your program statically in java. Just adding some lines into build.xml file and the problem is resolved.
<jar destfile="${jar.file}" basedir="${build.dir}" manifest="${manifest.file}">
<fileset dir="${classes.dir}" includes="**/*.class" />
<zipgroupfileset dir="${lib.dir}" includes="**/*.jar" />
</jar>
This codes indicates to ant that all jars referenced would be unzipped, adding the result into the new jar.
Further there are some tools as One-jar that put the whole jar, but you need special libraries. Here you can find a good discussion about this issue.
Hadoop
Hadoop is a framework that used map/reduce technique (developed by Google) to process large document collections. This framework is currently the most popular model to process text. It was develop by Apache and Yahoo.
But it has its opponents as this paper shows (here).
To configure hadoop in multi-node cluster, you can follow this blog entry (here). Is a very good tutorial.
Tika
Tika is a toolkit used to parse text documents as html, xml, doc, xls, or pdf. Tika is very simple and easy. For example, Tika can detect automatically which parse need for a specific file, and also can detect its charset. Right now I have to build a crawler, and I use Tika to parse html files.
to parse with Tika just you need the fallowing code:
InputStream input = new ByteArrayInputStream(mPage); String mimeType = new Tika().detect(input); metadata = new Metadata(); metadata.set(Metadata.CONTENT_TYPE, mimeType); DOMResult result = new DOMResult(); TransformerHandler transformerHandler = ((SAXTransformerFactory) SAXTransformerFactory.newInstance()).newTransformerHandler(); transformerHandler.setResult(result); new HtmlParser().parse(input, transformerHandler, metadata, new ParseContext());
I need a DOMtree, so I use TransformerHandler class as callback to save SAX events and convert into DOM nodes, but course you can use another handler. There are many handlers for specific proposes.
Drupal
Two days ago, my professor (and my boss) told me that I have to build a Web site for our FONDEF project. I have worked with Joomla, but a friend told me that there is another good toolkit called Drupal (http://drupal.org/). So, I installed Drupal 6 in my Ubuntu really easy with apt-get. The Ubuntu community also has a good tutorial in https://help.ubuntu.com/community/Drupal. Anyway, you can install Drupal in Ubuntu, following this simple steps:
Open a console, and update your apt-get, then try this.
sudo apt-get install drupal6
and then you have to restart the webserver Apache
sudo /etc/init.d/apache2 restart
And you’ll see the Drupal install pages in this URL:
http://localhost/drupal6/install.php
Simply follow the instructions, and you’ll have Drupal installed in your computer. (If you have some troubles visit http://drupal.org/node/439204.)
It’s really simple to build nice applications. There is a lot of modules and themes that allow us to build pro applications. Those modules are downloaded by means git. Git is a package for download things like wget.
Now I’m using the drupify template (http://drupal.org/node/237835) and also trying some modules like VotingAPI that help us to use a standardized API and schema for storing, retrieving, and tabulating votes for Drupal content (http://drupal.org/project/votingapi).
That’s all for now, anything new I’ll tell you soon.
Regards, Emir.
Useful chilean sites!
The ONG “Fundación Ciudadano Inteligente” FCI (Smart citizen foundation) is a national organization which pursuits bring useful information to chilean citizens. Right now, FCI has two important web sites which are described below:
- Vota inteligente (Vote smart): This site provides information about Chilean national congress activities, like current laws projects in process or senator and representative profiles.
- Acceso Inteligente (Smart access): Here you can find information about any government identity as Health department or public universities. Currently is a beta version.
I hope the sites will help to Chilean citizens to take smart decisions, especially for senators, representatives or presidents election.
Wellcome Pollo
Wellcome Pollo Pablo
Essential elements for Firefox
Firefox has many applications (plugins) that one can add to personalize it. This applications o plugins are divided in a lot of types. For example, you can connect to gmail or add a downloader manager, etc. This plugins are called addon which can be looked and downloaded from here
I left some addon examples very useful for firefox.
- You must install MR Tech Local Install, which you will can install complements for your system. So if you lost by any reason the firefox configuration, you could install it easily, and save your time. here
- For to see downloads status, there is a very good addon that shows your download at lower left corner, and doesn’t lunch that horrible windows. here
- To use your gmail account as a hard disk, you have one option to do it, this is downloading this complement to access your account through ftp. here
- There are moments that you can’t remember the translation of a word or phrase in other language, For that is this complement that translates to selected language. It is very useful. here
- Now to left firefox more “chulo”, You can search some skins. The best skins that I’ve found you can download. here
- Besides there are effects to chaning tabs. This you can download here
- If you wish access your FTP accounts, there is an addon to convert the browser in a FTP client. here
- You also can have a download manager for PDF files. This addon is very good. here
Those are the tips for now.
Reparar Firefox en DIINF
Para poder eliminar el mensaje que entrega firefox, relacionado a que ya esta corriendo en el sistema, el cual es “Firefox is already running, but is not responding. To open a new windows, you must first close the existing Firefox process, on restart your system.”
En windows:
Simplemente deben habilitar la opción de ver los archivos ocultos, luego buscar la carpeta .mozilla luego ir a la carpeta firefox y allí buscar una carpeta tipo xxxxxx.default en esta carpeta se encontrará un archivo de nombre parent.lock, solo deben eliminarlo, y abrir el firefox.
En Linux:
Deben abrir una terminal, acceder a la carpeta de firefox, tipeando: cd .mozilla/firefox , luego deben hacer un ls, y buscar una carpeta del tipo xxxxxx.default , deben acceder a esta y cambiar eliminar el archivo .parentlock con la instrucción rm .parentlock, si les entrega un error debe ser por los permisos de escritura, para cambiarlo deben tipear chmod +w .parentlock, y luego eliminarlo. Para finalizar deben devolver los permisos de escritura con la instrucción chmod -w .parentlock.
Espero les sirva.
Fuente: http://support.mozilla.com/
API de traducción de Google
El gran Google ha liberado una API de traducción, la cual está basada en su propio traductor, su utilización es muy sencilla, aca les va un ejemplo:
import com.google.api.translate.Language;
import com.google.api.translate.Translate;
public class Main {
public static void main(String[] args) {
try {
String translatedText = Translate.translate("Hola Mundo",
Language.SPANISH, Language.ENGLISH);
System.out.println(translatedText);
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
Como pueden ver es muy fácil de usar, pero por la experiencia de haber utilizado el traductor de Google, todos sabemos que este no es muy bueno con algunas frases, su motor de inteligencia es muy pobre. Incluso no toma en cuenta nuestra gran Ñ.
La API la puedes descargar para probar desde AQUÍ
Depinfi's Blog 