Introduction

The existing Java POJO domain models I needed to create jpa mappings for contained about 20-50 classes each.
Creating these mappings manually would have been a lot of work, so I looked into creating these automatically.
I did not expect to find a complete out-of-the-box solution, but I expected to find something that would enable me to avoid all the repetitive work.
However, I did not find any Java2JPA tool, so I wrote a simple mapping generator myself.

Generation of the mappings is not entirely automatable

There is probably no known Java2JPA tool because the mapping process is not entirely automatable.
Whereas popular ddl2jpa and jpa2java tools exist for most JPA Providers(Hibernate Tools for Hibernate, Eclipselink JPA Extensions,..), that provide complete “generate with the press of a button” output, java2jpa tools don’t exist probably because using them would usually require some manual work.
Human decisions need to be made while mapping.
For example, it is not possible to determine the field or combination of fields that represent the database id of a class by plain introspection, and it is also not possible to determine whether an abstract class should be mapped as a “mapped superclass” or as an entity that already has its own table.

The java2jpa tool code

The code is hosted at Github(java2jpa). The code can be downloaded as a zip. You can build the code with Maven.
The code is build around a few core classes: Java2JpaMappingGenerator, RenderJpaMappingForClassStrategy and JpaMappingRenderer.
The Java2JpaMappingGenerator is the class responsible for generating the mappings and has two important fields: a JpaMappingRenderer and a RenderJpaMappingForClassStrategy.
The RenderJpaMappingForClassStrategy is responsible for making the rendering decisions – e.g. which field will be mapped as id, how class inheritance will be mapped,.. – and the JpaMappingRenderer is responsible for rendering the actual files based on the rendering decisions made.
To use the tool efficiently more often than not a custom implementation of RenderJpaMappingForClassStrategy is required.

Basic usage of the code

In this basic usage example we will generate a jpa mapping for the following two pojo classes in the com.test.model.simple package: Student and Course.
In the unit tests of the source code an example with a more extended sample model is available.

public class Course {

  private Long id;
  private String title;
  private List<Student> students = new ArrayList<Student>();

  public Long getId() {
    return id;
  }
  public void setId(Long id) {
    this.id = id;
  }
  public String getTitle() {
    return title;
  }
  public void setTitle(String title) {
    this.title = title;
  }
  public List<Student> getStudents() {
    return students;
  }
  public void setStudents(List<Student> students) {
    this.students = students;
  }

}

public class Student {

  private int studentId;
  private String name;
  private Date birthDate;
  private Course course;

  public int getStudentId() {
    return studentId;
  }
  public void setStudentId(int studentId) {
    this.studentId = studentId;
  }
  public String getName() {
    return name;
  }
  public void setName(String name) {
    this.name = name;
  }
  public Date getBirthDate() {
    return birthDate;
  }
  public void setBirthDate(Date birthDate) {
    this.birthDate = birthDate;
  }
  public Course getCourse() {
    return course;
  }
  public void setCourse(Course course) {
    this.course = course;
  }
}

Make sure the java2jpa jar is available on the Java classpath.
Then execute the following code:

Java2JpaMappingGenerator java2JpaMappingGenerator =
     new Java2JpaMappingGenerator();
java2JpaMappingGenerator.setRenderJpaMappingForClassStrategy(
     new RenderJpaMappingForClassStrategyDefaultImpl());
JpaMappingRendererDefaultImpl jpaMappingRenderer =
     new JpaMappingRendererDefaultImpl("target/META-INF/orm.xml");
java2JpaMappingGenerator.setJpaMappingRenderer(jpaMappingRenderer);
java2JpaMappingGenerator.generateJpaMappingsForPackages("com.test.model.simple");
jpaMappingRenderer.createMappedFiles();

This will generate the following orm.xml in the target/META-INF folder:

<?xml version="1.0" encoding="UTF-8"?>
<entity-mappings version="2.0"
  xmlns="http://java.sun.com/xml/ns/persistence/orm"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/persistence/orm orm_2_0.xsd">
  <persistence-unit-metadata>
    <xml-mapping-metadata-complete/>
    <persistence-unit-defaults>
      <access>PROPERTY</access>
    </persistence-unit-defaults>
  </persistence-unit-metadata>
  <entity class="com.test.model.simple.Course">
    <table name="COURSE"/>
    <attributes>
      <id name="id">
        <generated-value strategy="AUTO"/>
      </id>
      <one-to-many mapped-by="course" name="students"/>
    </attributes>
  </entity>
  <entity class="com.test.model.simple.Student">
    <table name="STUDENT"/>
    <attributes>
      <id name="studentId">
        <generated-value strategy="AUTO"/>
      </id>
      <many-to-one fetch="LAZY" name="course">
        <join-column name="COURSE_ID"/>
      </many-to-one>
    </attributes>
  </entity>
</entity-mappings>

JpaMappingTester jpaMappingTester=new JpaMappingTesterOnHsqlImpl("TESTDB");
jpaMappingTester.test();

Implementing your own RenderJpaMappingForClassStrategy

For some models, RenderJpaMappingForClassStrategyDefaultImpl will work out of the box.
However, for most models a different implementation will be necessary.

The following table lists the assumptions this default implementation makes and mentions some override scenario’s.

Limitations of the tool

The xml mappings generated by the current implementation are far from complete.
Column level details are almost never specified in the generated mappings, only one inheritance mapping strategy is considered and collection cascades need to be added manually.
The tool only creates an initial mapping for each class, avoiding most of the repetitive work involved.

Stack Overflow

A while ago I asked on Stackoverflow whether a java2jpa tool already existed.
I answered my own answer now by pointing people to my own tool.
If the tool is of use to you, upvotes on that answer are appreciated.
Would be a nice community reward to get some reputation there out of contributing this.

OSGI

The Open Services Gateway Initiative (OSGI) makes it possible to break your application into multiple modules which can be independently deployed from each other, making it easier to manage cross-dependencies.
The different modules (coming in the form of bundles/jars for deployment) can be remotely installed, started, stopped, updated, and uninstalled without requiring a reboot.

OSGI Bundles

Each bundle is a tightly-coupled, dynamically loadable collection of classes, jars and configuration files that explicitly declares its external dependencies (if any).
Each bundle contains a detailed manifest MANIFEST.MF file, which includes different OSGI tags, and which is what physically makes an OSGI bundle different from a non-OSGI Java jar.

An example of such a manifest file:

Bundle-Name: Integrating Stuff OSGI test
Bundle-SymbolicName: com.integratingstuff.osgi
Bundle-Description: A Test OSGI bundle
Bundle-ManifestVersion: 2
Bundle-Version: 1.0.0
Bundle-Activator: com.integratingstuff.osgi.BundleActivator
Export-Package: com.integratingstuff.osgi;version="1.0.0"
Import-Package: org.osgi.framework;version="1.3.0"

Hence, building an OSGI jar instead of a regular jar comes down to setting the values in this manifest file correctly.

The first 5 tags in the example are obvious.

The 3 last ones are the more interesting ones:
Bundle-Activator: Indicates the class name to be invoked once a bundle is activated.
Export-Package: Expresses what Java packages contained in a bundle will be made available to the outside world.
Import-Package: Indicates what Java packages will be required from the outside world, in order to fulfill the dependencies needed in a bundle.

Activator

The Bundle-Activator BundleActivator class, declared in the manifest file, will be invoked once a bundle is activated. A typical OSGI jar, which contains services that should be registered with the OSGI container, contains such a Bundle-Activator declaration.
However, the declaration of a Bundle-Activator is optional. Some jars will only export certain packages and will not need a bundle activator/not register any services. The OSGI jar for the Spring framework, for example, does not require a Bundle Activator. All it does is exporting packages for other OSGI bundles to use.

An example of a typical BundleActivator:

public class Activator implements BundleActivator {

        public void start(BundleContext context) throws Exception {
                System.out.println("Starting: Integrating OSGI stuff");
                context.registerService(AccountService.class.getName(),
			new AccountServiceImpl(), null);
        }

        public void stop(BundleContext context) throws Exception {
                System.out.println("Stopping with integrating OSGI stuff");
                ServiceReference serviceReference = context.
			getServiceReference(AccountService.class.getName());
		if (serviceReference != null){
			context.ungetService(serviceReference);
		};
        }
}

For brevity, we do not include the AccountService class here. To compile this, you will also need the osgi-core lib on your classpath. If you want to experiment with the code, please download the Maven project made available in the last section of this article.

OSGI Lifecycle

The OSGI life cycle layer introduces dynamics that are normally not part of an application.

Every bundle within the OSGI container can be in one of the following states thanks to this mechanism:

Installed: The bundle has been successfully installed.
Resolved: All Java classes that the bundle needs are available. This state indicates that the bundle is either ready to be started or has stopped.
Starting: The bundle is being started, the BundleActivator.start method will be called, and this method has not yet returned. When the bundle has an activation policy, the bundle will remain in the Starting state until the bundle is activated according to its activation policy..
Active: The bundle has been successfully activated and is running; its Bundle Activator start method has been called and returned.
Stopping: The bundle is being stopped. The BundleActivator.stop method has been called but the stop method has not yet returned.
Uninstalled: The bundle has been uninstalled. It cannot move into another state.

OSGI containers

There are 3 popular OSGI container implementations: Knopflerfish, Equinox, and Apache Felix. Concierge seems to be receiving a lot of adaptation on mobile and embedded devices too, due to its limited file footprint of about 80 kBytes.

Adding the correct manifest to your bundle with Maven

To make sure Maven adds the correct manifest to our bundle, we use the org.apache.felix maven-bundle-plugin. This plugin generates the correct manifest for us, based on the configuration we pass, configured through Maven properties.
We also use the maven-dependency-plugin to be able to embed libraries and we configure the maven-jar-plugin to use the generated manifest file.

This is how our pom.xml looks like:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
    http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.integratingstuff</groupId>
  <artifactId>com.integratingstuff.osgi</artifactId>
  <version>1.0.0</version>
  <name>com.integratingstuff.osgi</name> 

  <build>
    <plugins>
      <!-- this is needed to enable reference in the Manifest file to Embedded libraries -->
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
          <execution>
            <id>copy-dependencies</id>
            <phase>compile</phase>
            <goals>
              <goal>copy-dependencies</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.3.2</version>
        <configuration>
          <archive>
            <manifestFile>${manifestFile}</manifestFile>
          </archive>
        </configuration>
      </plugin>
      <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <version>2.3.6</version>
        <extensions>true</extensions>
        <configuration>
          <manifestLocation>${manifestFolder}</manifestLocation>
          <excludeDependencies>${maven.bundleplugin.exclude.dependencies}</excludeDependencies>
          <instructions>
            <Bundle-SymbolicName>${project.build.bundle.symbolicname}</Bundle-SymbolicName>
            <Bundle-Name>${project.build.bundle.name}</Bundle-Name>
            <Bundle-Version>${project.build.bundle.version}</Bundle-Version>
            <Bundle-Activator>${project.build.bundle.activator}</Bundle-Activator>
            <Import-Package>${project.build.import.package}</Import-Package>
            <Export-Package>${project.build.export.package}</Export-Package>
            <Private-Package>${project.build.private.package}</Private-Package>
            <Web-ContextPath>${project.build.web.contextpath}</Web-ContextPath>
            <Webapp-Context>${project.build.web.contextpath}</Webapp-Context>
            <Bundle-ClassPath>${project.build.bundle.classpath}</Bundle-ClassPath>
            <Embed-Dependency>${project.build.embed.dependency}</Embed-Dependency>
            <Embed-Directory>${project.build.embed.directory}</Embed-Directory>
            <Embed-Transitive>${project.build.embed.transitive}</Embed-Transitive>
          </instructions>
          <supportedProjectTypes>
            <supportedProjectType>jar</supportedProjectType>
            <supportedProjectType>bundle</supportedProjectType>
            <supportedProjectType>war</supportedProjectType>
          </supportedProjectTypes>
        </configuration>
        <executions>
          <execution>
            <id>bundle-manifest</id>
            <phase>process-classes</phase>
            <goals>
              <goal>manifest</goal>
            </goals>
          </execution>
        </executions>

      </plugin>
    </plugins>
  </build>

  <properties>
    <project.build.bundle.symbolicname>${project.artifactId}</project.build.bundle.symbolicname>
    <project.build.bundle.name>${project.name}</project.build.bundle.name>
    <project.build.bundle.version>${project.version}</project.build.bundle.version>
    <project.build.bundle.activator></project.build.bundle.activator>
    <project.build.import.package>*</project.build.import.package>
    <project.build.export.package>*</project.build.export.package>
    <project.build.private.package></project.build.private.package>
    <project.build.bundle.classpath></project.build.bundle.classpath>
    <project.build.web.contextpath></project.build.web.contextpath>
    <project.build.embed.dependency></project.build.embed.dependency>
    <project.build.embed.directory></project.build.embed.directory>
    <project.build.embed.transitive></project.build.embed.transitive>
    <manifestFolder>src/main/resources/META-INF</manifestFolder>
    <manifestFile>${manifestFolder}/MANIFEST.MF</manifestFile>
  </properties>

</project>

Testing OSGI bundles with Spring

When building OSGI bundles, it is probably a good idea to check whether the built bundles are valid. We are going to check this by writing unit tests that extend from AbstractConfigurableBundleCreatorTests, from the spring-osgi-mock jar. This test will start an OSGI container and run the test in it. Tests like these can catch OSGI dependency problems automatically, requiring no manual deploy, allowing easy OSGI integration testing.

To start, add the following dependencies section to the Maven pom.xml:

<dependencies>
  <!-- test dependencies -->
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>spring-osgi-core</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>spring-osgi-mock</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>spring-osgi-test</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>spring-osgi-annotation</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>spring-osgi-extender</artifactId>
    <version>1.2.1</version>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>org.apache.felix</groupId>
    <artifactId>org.osgi.core</artifactId>
    <version>1.4.0</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.felix</groupId>
    <artifactId>org.apache.felix.framework</artifactId>
    <version>1.7.0-private</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.felix</groupId>
    <artifactId>org.apache.felix.main</artifactId>
    <version>1.7.0-private</version>
    <scope>test</scope>
  </dependency>

  <!-- from http://dev.anyframejava.org/maven/repo/org/springframework/osgi/log4j.osgi/1.2.15-SNAPSHOT/ -->
  <dependency>
    <groupId>org.springframework.osgi</groupId>
    <artifactId>log4j.osgi</artifactId>
    <version>1.2.15-SNAPSHOT</version>
    <scope>test</scope>
  </dependency>
</dependencies>

This will add the spring osgi dependencies we need.

This will also add all the dependencies needed to run the Felix OSGI container, which will be the OSGI container we will run our tests in.

Note: You might need to download and install the log4j.osgi manually from http://dev.anyframejava.org/maven/repo/org/springframework/osgi/log4j.osgi/1.2.15-SNAPSHOT into your own nexus or local maven repo.

Then add the following test:

public class LoadOSGITestCase extends AbstractConfigurableBundleCreatorTests{

	@Override
	protected String[] getTestBundlesNames() {
		return new String[] { "com.integratingstuff, com.integratingstuff.osgi, 1.0.0" };
	}

	public void testOSGIPlatformStarts() throws Exception {
		System.out.println(bundleContext.getProperty(Constants.FRAMEWORK_VENDOR));
		System.out.println(bundleContext.getProperty(Constants.FRAMEWORK_VERSION));
		System.out.println(bundleContext.getProperty(Constants.FRAMEWORK_EXECUTIONENVIRONMENT));
	}

	public void testOSGIEnvironment() throws Exception {
		Bundle[] bundles = bundleContext.getBundles();
		for (int i = 0; i < bundles.length; i++) {
			System.out.print(OsgiStringUtils.nullSafeName(bundles[i]));
			System.out.print(", ");
		}
	}

	public void testRetrieveService() throws Exception {
		Properties props = new Properties();
		props.put("Language", "English");

		ServiceTracker accountServiceTracker = new ServiceTracker(bundleContext, AccountService.class.getName(), null);
		accountServiceTracker.open();
		AccountService accountService = (AccountService) accountServiceTracker.getService();
		accountService.createAccount(1000l);
	}

	@Override
	protected String getPlatformName() {
		return Platforms.FELIX;
	}

}

And run it.

These tests will run within the Felix OSGI container. The two first tests just write out some info about the OSGI environment itself. The last test fetches a reference to this service and then calls a method on it.

Troubleshooting OSGI dependency problems

When starting with OSGI, you need to get used to not having a global classpath, but instead, take into account which packages a bundle exports or imports.

You might run into problems because of this. In what follows, we cover some common import/export package mistakes. Different OSGI containers can have different implementations, so the error messages you see because of these mistakes might differ a bit. One OSGI container might fail to resolve a bundle because of a conflict, while another one might just load one of two conflicting bundles and report a ClassNotFoundException when a class residing in a discarded or unknown bundle is used. In certain containers, it is possible to get away with making one of the mistakes covered here, in certain circumstances. Still, they are errors against the specification and should always be avoided.

If you want to experiment with these errors to better understand them, please download this OSGI test project. It is in a correct starting state, but in what follows, we will indicate how to produce every covered error using this project.
Any of these errors will cause the unit test in the com.integratingstuff.osgi.test module to fail.

Failing to export a package needed by another bundle

This error can be produced by changing the project.build.export.package Maven property in the pom.xml of the com.integratingstuff.osgi.services1 module to “com.test.model” instead of “*”. Then com.test.service of the com.integratingstuff.osgi.services1 module will not be exported then(not declared in the Export-Package section of the manifest), and the AccountService class will not be available to other modules that need it, such as the com.integratingstuff.osgi.services2 module.

Failing to import a required package

This error can be produced by changing the project.build.import.package Maven property in the pom.xml of the com.integratingstuff.osgi.services2 module to “com.test.model” instead of “*”. Then com.test.service of the com.integratingstuff.osgi.services1 module will not be imported then(not declared in the Import-Package section of the manifest), and the AccountService class will be unknown to the com.integratingstuff.osgi.services2 module.

Exporting conflicting packages

This error can be produced by renaming the com.test.person.service package in the com.integratingstuff.osgi.services2 module to com.test.service, so it conflicts with the com.test.service package of the com.integratingstuff.osgi.services1 module.

Depending on the container, bundles importing the conflicting package will not resolve(“package uses conflict” errors) or only one of the two conflicting packages will be known by the container, resulting in ClassNotFoundExceptions.

In an OSGI environment, two bundles should not export a package with the same name.

It is recommended that readers already have some experience with MongoDb and Amazon EC2.

Introduction

What is MongoDB?

MongoDB is an open source document-oriented NoSQL database.

Instead of storing data in tables as is done in a “classical” relational database, MongoDB stores structured data as JSON-like documents with dynamic schemas(“BSON”).

Although earlier versions of MongoDB were criticized, “MongoDB is Web scale“, these issues were resolved(for example, journaling is enabled by default now). MongoDb is now mature and production ready, and it is used extensively by MTV, Craigslist and Foursquare among others.

Whether you should use it or not depends. This Dzone Reference Card on Getting Started with NoSql might help. It advises to base the choice around Brewer’s CAP Theorem: “It’s impossible for a distributed computer system to simultaneously provide Consistency, Availability and Partition Tolerance”. Decide which 2 of these you need most, and you know what type of database you need.

MongoDB offers a lot of the features RDBMS databases offer, such as adding indices to and querying on all fields. Some NoSql databases/services, such as Amazon DynamoDB lack this, which makes their use rather limited.

However, MongoDB does not enforce consistency. You can put a reference(DBRef) in a record in collection1(a MongoDB “collection” is the equivalent of a RDBMS table) to a record in collection2 and then delete the record in collection2. No error would be thrown. The reference would still be there, but when someone would try to resolve it, no record in collection2 would be found.

What MongoDB offers in return however, is easy replication and sharding, which makes it excel at Availability and Partition Tolerance. Unlike RDBMS databases, which are usually scaled vertically(“making the machine bigger”), MongoDB can be easily scaled horizontally( “adding more machines”).

What is Amazon EC2?

The Amazon Elastic Compute Cloud(Amazon EC2) is a central part of Amazon’s cloud computing platform Amazon Web Services(AWS).
Users can rent virtual computers on EC2 to which they get full root access.
They pay by the hour for active servers, hence the term “elastic”. EC2 makes scalable deployment of applications easy by providing a web service through which users can boot instances by providing a machine image(for example, a Linux Image on which Apache/Php/Wordpress/Mysql or Java/Tomcat is already installed).

Although using Amazon EC2 is not necessarily less expensive than buying your own hardware, it is a good choice for most startups, since instances can be easily added or removed, or replaced by larger or smaller ones. You can scale your costs with your load, and thus, you are protected against overinvestment and you are prepared for scaling up in a matter of minutes in case you get a traffic peak.

Many startups, such as Foursquare and BitBucket, run entirely on Amazon EC2/AWS.

What is Amazon CloudFormation?

There are 3 interfaces through which EC2 instances/AWS Resources can be set up. The first one is the web interface, which provides an intuitive graphical interface. The second one consists of the EC2 Command Line Tools, which make it possible to launch and manage instances through scripting and which are used by third party AWS tools such as the AWS/EC2 Plugin for Eclipse.

The third one is CloudFormation. With Amazon CloudFormation, you describe AWS resources in a template. You then instruct Amazon CloudFormation to read this template and initialize the resources associated with it.

If you’ve never used CloudFormation before, you should go through its Getting Started Guide. We will only discuss the basic layout of a Cloudformation template file here briefly. There are 4 sections that appear in almost every template. We will use snippets of a Tomcat 7 template as examples.

Parameters

When a template is loaded, CloudFormation prompts the user to enter a value for every parameter that is defined in the Parameters section.

An example Parameters section:

"Parameters" : {
        "KeyName" : {
            "Description" : "Name of an existing EC2 KeyPair to enable SSH access",
            "Type" : "String"
        },

        "InstanceType" : {
            "Type" : "String",
            "Default" : "m1.large",
            "AllowedValues" : [ "t1.micro", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.xlarge", "cc1.4xlarge" ],
            "Description" : "EC2 instance type (e.g. t1.micro, m1.small, m1.medium, m1.large, m1.xlarge, m2.xlarge)"
        },

        "SecurityGroupName" : {
            "Description" : "Security group name for the instance",
            "Type" : "String"
        }
}

With the above example, the user will be prompted for the name of a ssh keypair, the type of the EC2 instance that will be created by the template and the name of the security group to which this instance will belong.

Mappings

In the Mappings section, mappings of certain values to other values are described.

An example Mappings section:

"Mappings" : {
        "RegionImageZone" : {
            "us-east-1"      : { "64" : "ami-e565ba8c"},
            "us-west-2"      : { "64" : "ami-3ac64a0a"},
            "us-west-1"      : { "64" : "ami-e78cd4a2"},
            "eu-west-1"      : { "64" : "ami-f9231b8d"},
            "ap-southeast-1" : { "64" : "ami-be3374ec"},
            "ap-northeast-1" : { "64" : "ami-e47acbe5"},
            "sa-east-1"      : { "64" : "ami-a6855bbb"}
        }
}

The above mapping could be used in a template to determine the code for the AMI(Amazon Machine Image) to be used for a certain region. The above mappings point to the basic Amazon Linux AMI for different regions. Since the template knows on which region it is executing, providing and using a mapping like this is usually a good idea.

Resources

This is the most important section of the template and describes the resources to initialize. EC Instances are typically seen here, but other kinds of AWS Resources can be defined as well, such as security rules and S3(Amazon Simple Storage Service, often used extensively with EC2 Instances) resources.

In our example, the only resource associated with the template is an EC2 instance on which a Tomcat 7 server will be installed:

"Resources" : {
        "Tomcat7" : {
            "Type" : "AWS::EC2::Instance",
            "Metadata" : {
                "AWS::CloudFormation::Init" : {
                    "config" : {
                        "packages" : {
                            "yum" : {
                                "apr-devel" : [],
                                "openssl-devel" : [],
                                "gcc" : [],
                                "java-1.6.0-openjdk-devel" : [],
                                "tomcat7": []
                            }
                        }
                    }
                }
            },

            "Properties" : {
                "InstanceType" : { "Ref" : "InstanceType" },
                "ImageId" : { "Fn::FindInMap" : [ "RegionImageZone", { "Ref" : "AWS::Region" }, "64" ] },
                "SecurityGroups" : [ { "Ref" : "SecurityGroupName" } ],
                "KeyName" : { "Ref" : "KeyName" },
                "Tags" : [
          			{"Key" : "Name", "Value" : "Tomcat7" }
        		]}
	}
}

Notice how both the Parameters (with “Ref”) and the Mappings(with “Fn::FindInMap”) section are used to init the instance.

Outputs

The Outputs of the template. This could be the public ip of the created instance.

An example outputs section:

"Outputs" : {
        "InstanceName" : {
            "Value" : { "Fn::GetAtt" : [ "Tomcat7", "PublicDnsName" ] },
            "Description" : "public DNS name of the new Tomcat7"
        }
    }

Outputs are visible in the “Outputs” tab of the CloudFormation web interface:

The CloudFormation templates for setting up a MongoDb replicaset with an Arbiter

What is a MongoDb replicaset?

A MongoDb replicaset is a form of asynchronous master/slave replication, adding automatic failover and automatic recovery of member nodes. It is heavily recommended to never deploy MongoDb to one standalone machine, but always to replicasets, in sets of 3 or a higher uneven number of machines. A normal full replicaset features 3 nodes that store data. In a replicaset with 2 nodes and an arbiter, only the first 2 nodes store data. The master only replicates the data to the one slave. The arbiter is only there in case a new master needs to be elected. It is necessary to prevent MongoDb to go in read-only mode if one of the two other nodes goes down.

The 10gen templates vs our templates

10gen, the MongoDB company, already has extensive documentation on how to setup MongoDB on EC2, including documentation on how to setup a replicaset on EC2, and even provides Cloudformation templates to setup a replicaset.

Our templates, which describe a (more budget) MongoDb replicaset consisting of 2 normal nodes and an arbiter as opposed to a replicaset consisting of 3 normal nodes – are heavily based on these 10gen Cloudformation templates.

We did make some changes to these templates which we consider improvements though:

• Instead of 4 S3 volumes in RAID 10 per node, our template inits 8 S3 volumes in RAID 10 per node. This is what the 10gen documentation recommends.
• The nodiratime directive was added to all mounts in /etc/fstab. This is also what the 10gen documentation recommends.
• File descriptor limits were raised in /etc/security/limits.conf(“* hard nofile 100000″ and “* soft nofile 100000″) Raising file descriptor limits is also recommended by 10gen for production deployments.

We also made the following changes. These are not necessarily improvements:

• Security group definitions are not included in our templates, so you need to create these manually yourself before you can use our templates. Although this requires a bit more EC2 Knowledge, you probably don’t want to tie these security groups to only this Cloudformation stack. If you run the template again later, you probably want to put the new instances in the same security group again, not create a new one with the same security rules.
• Although 10gen recommends to not use anything smaller than a large instance(m1.large) for MongoDB production deployments, large instances are expensive to test on. Therefore, we made it possible to select the Micro(t1.micro) and Small(m1.small) instance types for the normal MongoDb nodes as well.

The CloudFormation templates for a MongoDb replicaset with an Arbiter

IMPORTANT! Remember that running these templates on CloudFormation will create AWS resources and Amazon will bill you for AWS resource usage! These templates will create EC2 instances, S3 volumes and some IAM security resources. When t1.micro instances are chosen, it should be possible to test these templates entirely on the Amazon AWS Free Tier, but we do not guarantee this.

The templates:

• ReplicaSetWithArbiter.template: This template will first initialize a slave and an arbiter instance by using the templates below, and then init the master instance.
• MongoDbServer.template: This template describes the instance that will be configured as the slave of the replicaset. When run seperately, this creates a standalone MongoDb server.
• Arbiter.template: This template describes the Arbiter instance.

To setup the replicaset, it is enough to provide the first url to CloudFormation. It will call the other two templates itself.

If you are heading towards a production MongoDB deployment, it is probably best to download and host these templates yourself. In that case, you will need to change the TemplateUrl of the Arbiter and the Secondary instance in the ReplicaSetWithArbiter.template file to your own url.

A CloudFormation template for the Mongo Monitoring Service

The Mongo Monitoring Service(MMS) enables you to proactively monitor your MongoDb cluster. First, you set up an mms agent that has access to your MongoDb nodes. This agent will then fetch metrics from your mongos and send these to 10gen servers, which will show you different graphs about your MongoDb deployment.

First, you will need to sign up for this monitoring service through http://www.10gen.com/mongodb-monitoring-service. After signing up, you can retrieve your API key and Secret key, both necessary to setup the agent.

You can then use the following CloudFormation template to set up the mms agent on an EC2 Micro instance:

MongoMonitoringServer.template

After setting up this Micro instance, you only need to add the hosts(in case of a replicaset, at least one ip to a mongo node; mms will find the other nodes) through the mss web interface, as explained in its documentation.

I used to post a lot of articles about using the webservice APIS of third party sites on this blog. This is going to be another post like that.
This post describes how to use the Java Wikipedia API to fetch and format the contents of a Wikipedia article.

The Wikipedia API

We cover a basic use case: getting the contents of the “Web service” article.

To fetch the contents for this article, the following url suffices:

http://en.wikipedia.org/w/api.php? format=xml&action=query&titles=Web%20service&prop=revisions&rvprop=content

A request to this url will return an xml document which includes the current wiki markup for the page titled “Web service”. As the request parameters indicate, these requests are highly configurable. For example, other formats than xml, such as json, are possible. For a full list of available parameters, visit http://en.wikipedia.org/w/api.php.

We are not going to construct these urls ourselves. We are going to use bliki, the Java wikipedia API library instead.

Getting the Java Wikipedia API lib

If you are using Maven you need to add the following repository to your pom:

<repository>
  <id>info-bliki-repository</id>
  <url>http://gwtwiki.googlecode.com/svn/maven-repository/</url>
  <releases>
    <enabled>true</enabled>
  </releases>
</repository>

together with the following dependency:

<!-- bliki -->
<dependency>
  <groupId>info.bliki.wiki</groupId>
  <artifactId>bliki-core</artifactId>
  <version>3.0.17</version>
</dependency>

and if you want the addons:

<dependency>
  <groupId>info.bliki.wiki</groupId>
  <artifactId>bliki-addons</artifactId>
  <version>3.0.17</version>
</dependency>

If you are not using Maven, just grab the jar from the linked project page.

Usage examples of this lib are at http://code.google.com/p/gwtwiki/wiki/HTML2Mediawiki.

Basic example: getting the contents of an article

However, the basic usage example given in the documentation, at this time, does not compile with the current version of the lib.
Therefore, we will start with a basic usage example of which no variant is listed there and extend this example.

We are going to list the code to fetch the content of the “Web service” page and render it as html. Note that to get a specific page, you need to know its title.
If the page does not exist, a result with one empty page will be returned.
For ambiguous titles, the disambiguation page will be given too, so even if you get a non-empty result, you still need to check it thoroughly.

String[] listOfTitleStrings = { "Web service" };
User user = new User("", "", "http://en.wikipedia.org/w/api.php");
user.login();
List<Page> listOfPages = user.queryContent(listOfTitleStrings);
for (Page page : listOfPages) {
  WikiModel wikiModel = new WikiModel("${image}", "${title}");
  String html = wikiModel.render(page.toString());
  System.out.println(html);
}

We are instantiating a user on the English wikipedia endpoint. Since we are only going to read, we can login anonymously.
We query the english Wikipedia for the specified titles and get one page as result in the listOfPages variable.
We then instantiate a WikiModel. This class will render the html and its constructor parameters – imageBaseUrl and linkBaseUrl – determine where the rendered images and links will point too. For example, if you want these to point to local files, you would supply a local path. In the example, I made it completely relative. In the official documentation, these are “http://www.mywiki.com/wiki/${image}” and “http://www.mywiki.com/wiki/${title}”, which you would use if you were putting a Wikipedia copy at http://www.mywiki.com/wiki/.
We then render the page as html and print it out to the console.

The outputted rendering is very rudimentary and is far from complete though:

• Wikipedia magic variables, recognizable by their {{…}} markup, are not rendered. Instead, they are displayed literally.
• By default, all markup is rendered. However, you might need to leave certain parts out or modify the content a bit before it is displayed for your particular use case.

Handling magic variables

Most magic words are not supported by the Java Wikipedia API.
We need to implement their rendering ourselves.
If you want to do some advanced converting of the Wikipedia content, such as handling these magic words, you need to extend the WikiModel class. More info about this is at http://code.google.com/p/gwtwiki/wiki/Mediawiki2HTML.

This is what we are doing here:

package com.integratingstuff.wikimedia;

import java.util.Locale;
import java.util.Map;
import java.util.ResourceBundle;

import info.bliki.wiki.model.Configuration;
import info.bliki.wiki.model.WikiModel;
import info.bliki.wiki.namespaces.INamespace;

public class MyWikiModel extends WikiModel{

  public MyWikiModel(Configuration configuration, Locale locale,
    String imageBaseURL, String linkBaseURL) {
      super(configuration, locale, imageBaseURL, linkBaseURL);
  }
  public MyWikiModel(Configuration configuration,
    ResourceBundle resourceBundle, INamespace namespace,
    String imageBaseURL, String linkBaseURL) {
      super(configuration, resourceBundle, namespace, imageBaseURL, linkBaseURL);
  }
  public MyWikiModel(Configuration configuration, String imageBaseURL,
    String linkBaseURL) {
      super(configuration, imageBaseURL, linkBaseURL);
  }
  public MyWikiModel(String imageBaseURL, String linkBaseURL) {
    super(imageBaseURL, linkBaseURL);
  }

  @Override
  public String getRawWikiContent(String namespace, String articleName,
    Map<String, String> templateParameters) {
      String rawContent = super.getRawWikiContent(namespace, articleName, templateParameters);

      if (rawContent == null){
        return "";
      }
      else {
        return rawContent;
      }
    }
}

The overriden getRawWikiContent in the above MyWikiModel code returns null for most magic words in its default implementation. A magic word such as {{InfoBox}} would pass through this code with the default namespace=”Template” and articleName=”InfoBox”. If null is returned, the magic word will be outputted in the rendered html as is(so for {{InfoBox}}, this would be {{InfoBox}}). So, the resulting html is full of these unreadable tags, which does not make it look pretty printed.
What we are doing in the above code to solve this is returning “” instead of null, so the magic word does not get rendered at all.
Nothing is stopping you from returning something else though.

Controlling the rendering of the html by implementing an ITextConverter

For my particular use case, I also did not want to render any html links, I did not want to render any references and I did not want to render any images. The WikiModel class does not implement support for leaving out these things. However, the overloaded render method of WikiModel can take an ITextConverter object as an argument, the object that is responsible for converting the parsed nodes to html(or another format, like pdf or plain text). The default ITextConverter, used when none is specified as an argument, is HTMLConverter, with its property noLinks set to false by default.
However, there is a HTMLConverter constructor which sets the noLinks boolean. By passing true to this constructor, no links will be rendered. Their content will be rendered as plain text instead.
Since I still had to leave out the reference and image elements, I still ended up subclassing the HTMLConverter.
First, I made a more extensible version of it:

package com.ceardannan.exams.wikimedia;

import info.bliki.htmlcleaner.ContentToken;
import info.bliki.htmlcleaner.EndTagToken;
import info.bliki.htmlcleaner.TagNode;
import info.bliki.htmlcleaner.Utils;
import info.bliki.wiki.filter.HTMLConverter;
import info.bliki.wiki.model.Configuration;
import info.bliki.wiki.model.IWikiModel;
import info.bliki.wiki.model.ImageFormat;
import info.bliki.wiki.tags.HTMLTag;

import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

/**
 * A converter which renders the internal tree node representation as specific
 * HTML text, but which is easier to change in behaviour than its superclass and
 * has a noImages property, which can be set to leave out all images
 *
 */
public class ExtendedHtmlConverter extends HTMLConverter {

  private boolean noImages;

  public ExtendedHtmlConverter() {
    super();
  }

  public ExtendedHtmlConverter(boolean noLinks) {
    super(noLinks);
  }

  public ExtendedHtmlConverter(boolean noLinks, boolean noImages) {
    this(noLinks);
    this.noImages = noImages;
  }

  protected void renderContentToken(Appendable resultBuffer,
      ContentToken contentToken, IWikiModel model) throws IOException {
    String content = contentToken.getContent();
    content = Utils.escapeXml(content, true, true, true);
    resultBuffer.append(content);
  }

  protected void renderHtmlTag(Appendable resultBuffer, HTMLTag htmlTag,
      IWikiModel model) throws IOException {
    htmlTag.renderHTML(this, resultBuffer, model);
  }

  protected void renderTagNode(Appendable resultBuffer, TagNode tagNode,
      IWikiModel model) throws IOException {
    Map<String, Object> map = tagNode.getObjectAttributes();
    if (map != null && map.size() > 0) {
      Object attValue = map.get("wikiobject");
      if (!noImages) {
        if (attValue instanceof ImageFormat) {
          imageNodeToText(tagNode, (ImageFormat) attValue, resultBuffer, model);
        }
      }
    } else {
      nodeToHTML(tagNode, resultBuffer, model);
    }
  }

  public void nodesToText(List<? extends Object> nodes,
      Appendable resultBuffer, IWikiModel model) throws IOException {
    if (nodes != null && !nodes.isEmpty()) {
      try {
        int level = model.incrementRecursionLevel();

        if (level > Configuration.RENDERER_RECURSION_LIMIT) {
          resultBuffer
              .append("<span class=\"error\">Error - recursion limit exceeded rendering tags in HTMLConverter#nodesToText().</span>");
          return;
        }
        Iterator<? extends Object> childrenIt = nodes.iterator();
        while (childrenIt.hasNext()) {
          Object item = childrenIt.next();
          if (item != null) {
            if (item instanceof List) {
              nodesToText((List) item, resultBuffer, model);
            } else if (item instanceof ContentToken) {
              // render plain text content
              ContentToken contentToken = (ContentToken) item;
              renderContentToken(resultBuffer, contentToken, model);
            } else if (item instanceof HTMLTag) {
              HTMLTag htmlTag = (HTMLTag) item;
              renderHtmlTag(resultBuffer, htmlTag, model);
            } else if (item instanceof TagNode) {
              TagNode tagNode = (TagNode) item;
              renderTagNode(resultBuffer, tagNode, model);
            } else if (item instanceof EndTagToken) {
              EndTagToken node = (EndTagToken) item;
              resultBuffer.append('<');
              resultBuffer.append(node.getName());
              resultBuffer.append("/>");
            }
          }
        }
      } finally {
        model.decrementRecursionLevel();
      }
    }
  }

  protected void nodeToHTML(TagNode node, Appendable resultBuffer,
      IWikiModel model) throws IOException {
    super.nodeToHTML(node, resultBuffer, model);
  }

}

The functionality of the above HTMLConverter is almost the same as the original one, but the code is divided into more methods, for easier overriding, and a noImages boolean is added as well, which leaves out all the images at render time if set to true.

And then I subclassed this class like this:

package com.ceardannan.exams.wikimedia;

import info.bliki.htmlcleaner.ContentToken;
import info.bliki.htmlcleaner.Utils;
import info.bliki.wiki.model.IWikiModel;
import info.bliki.wiki.tags.HTMLTag;

import java.io.IOException;

public class MyHtmlConverter extends ExtendedHtmlConverter {

  public MyHtmlConverter() {
    super();
  }

  public MyHtmlConverter(boolean noLinks) {
    super(noLinks);
  }

  public MyHtmlConverter(boolean noLinks, boolean noImages) {
    super(noLinks, noImages);
  }

  protected void renderContentToken(Appendable resultBuffer,
      ContentToken contentToken, IWikiModel model) throws IOException {
    String content = contentToken.getContent();
    content = content.replaceAll("\\(,", "(").replaceAll("\\(\\)", "()");
    content = Utils.escapeXml(content, true, true, true);
    resultBuffer.append(content);
  }

  protected void renderHtmlTag(Appendable resultBuffer, HTMLTag htmlTag,
      IWikiModel model) throws IOException {
    String tagName = htmlTag.getName();
    if ((!tagName.equals("ref"))) {
      super.renderHtmlTag(resultBuffer, htmlTag, model);
    }
  }
}

If the converter encounters a “ref” html tag, it does not render the html tag. This results in no references getting rendered at all.
I also changed the rendering of the content a bit. This is because returning of “” for the magic words(see above), might leave (, or () in the text, and the line that replaces these cleans up the rendered html.

The code that we call to get the html is now:

MyWikiModel wikiModel = new MyWikiModel("${image}", "${title}");
String currentContent = page.getCurrentContent();
String html = wikiModel.render(
  new MyHtmlConverter(true, true), currentContent);

The end result

We now have a properly formatted, completely offline article, with all its external links, images and references stripped:

This article is for people who played around in XCode and objectiveC already, and are familiar with the basic iOS UIView and related classes.

Thanks!

The project these tips are applied to are my language classes for iPhone and iPad, such as Spanish Class, French Class and German Class. If you want to thank me for this article, please download a free version, and send me your feedback or give it a 4 or 5 star rating if you like the app.

Tip 1 – UIView: drawing an image as a background

Although you can not set an image as the background of a UIView in an interface builder xib file, it is easy to load the image as a color and assign it as the backgroundcolor to a view:

UIView *mainView = self.view;
mainView.backgroundColor = [UIColor colorWithPatternImage:
     [UIImage imageNamed: @"languages-menu.png"]];

Which results in the following background for the main menu UIView of the languages app:

Tip 2 – UIButton: fixing the position of the image and the label within

As shown in the screenshot above, I wanted to put the image and text within a main menu button at a specific location, so the images in the 10 different buttons would appear symmetric to each other.

This is very easy to do with the following method:

- (void) fixButton:(UIButton *) button{
    //positioning
    UIImageView *imageView = button.imageView;
    UILabel *label = button.titleLabel;

    CGRect imageFrame = imageView.frame;
    CGRect labelFrame = label.frame;

    imageFrame.origin.x = 10;
    labelFrame.origin.x = 10 + imageFrame.size.width + 5;

    imageView.frame = imageFrame;
    label.frame = labelFrame;
}

We are changing the x values of the origin of the frames of the two views within the button, making the image start at x coordinate 10 within the UIButton, and putting the label next to it, with only 5 pixels in between.

The key thing to realize here is that every view – UIView and subclasses like UITableView, UIButton,… – are positioned on the screen the same way: they have a frame, with an origin with an x and an y coordinate and a size with a width and a length. Changing their values is easy, and not considered bad practice.
Unlike as for Android, where you want to use relative layouts to accomodate for the wide range of possible screen sizes and shapes, playing with these values is common practice on iOS, since you know everything about the screensize. There are only 4 possible resolutions anyway: landscape and portrait, both for iPhone and iPad.

Tip 3 – UITableView: clearing the background of a table

In the exercises screens, I wanted to present the questions on a schoolboard interface, as if the questions and possible answers were written on the schoolboard. The list of possible answers is wrapped in a UITableView, so it is easy to change the number of answers(they used to be presented as a UIPicker, but that just looked awful, and there is no way to achieve the below effect with a picker anyway).

To do this, I had to manually clear the background of the tableview, which I implemented as an Objective-C category:

#import "UITableView+RemoveBackgroundFromTable.h"

@implementation UITableView (RemoveBackgroundFromTable)

-(void) removeBackgroundFromTable{
    self.backgroundColor = [UIColor clearColor];
    self.opaque = NO;
    self.backgroundView = nil;
}

@end

The first two lines can be set in a xib file, by unchecking “opaque” and by setting the color of the UITableView to “clearColor”. The last line cannot be done in a xib file though and is necessary as well. Not only the backgroundColor needs to be set to transparant, the backgroundView of the UITableView needs to be set to nil as well.

Tip 4 – UITableview: showing cells with dynamic height

When doing iOS development, you will want to tamper with the height of cells sooner or later. The UITableViewCells do not adjust their height automatically, and when the cell spans a few sentences, you cannot set the height in advance, because you just dont know it.

Credit were credit is due, this post describes exactly how to resize cells to accommodate different cell heights. However, since I was working with my own UITableViewCell subclasses which usually had another view to the left, I had to rewrite the StringHelper category a bit and add an xOffset:

@implementation NSString (StringHelper)

- (CGFloat)RAD_textHeightForSystemFontOfSize:(CGFloat)size xOffset:(int) xOffset{
    //Calculate the expected size based on the font and linebreak mode of the label
    CGFloat maxWidth = [UIScreen mainScreen].bounds.size.width - 25 - xOffset;
    CGFloat maxHeight = 9999;
    CGSize maximumLabelSize = CGSizeMake(maxWidth,maxHeight);

    CGSize expectedLabelSize = [self sizeWithFont:[UIFont systemFontOfSize:size] constrainedToSize:maximumLabelSize lineBreakMode:UILineBreakModeWordWrap];

    return expectedLabelSize.height;
}

- (CGRect)RAD_frameForCellLabelWithSystemFontOfSize:(CGFloat)size xOffset:(int) xOffset{
    CGFloat width = [UIScreen mainScreen].bounds.size.width - 25 - xOffset;
    CGFloat height = [self RAD_textHeightForSystemFontOfSize:size xOffset:xOffset] + 10.0;
    return CGRectMake(10.0f+xOffset, 10.0f, width, height);
}

- (void)RAD_resizeLabel:(UILabel *)aLabel WithSystemFontOfSize:(CGFloat)size xOffset:(int) xOffset{
    aLabel.frame = [self RAD_frameForCellLabelWithSystemFontOfSize:size xOffset:xOffset];
    aLabel.text = self;
    [aLabel sizeToFit];
}

@end

To make sure the cell resizes properly, it is then only necessary to resize the UILabel in your tableView: cellForRowAtIndexPath: method:

- (UITableViewCell *)tableView:(UITableView *)tableView
		 cellForRowAtIndexPath:(NSIndexPath *)indexPath {
        CustomCell *cell = [self getCustomCell];
        NSUInteger row = [indexPath row];

        NSString *element = [list objectAtIndex:row];

        cell.customLabel.numberOfLines = 0;
        [element RAD_resizeLabel:cell.inTargetLanguage WithSystemFontOfSize:17 xOffset:40.0];

        CGRect secondFrame = cell.customLabel.frame;
        secondFrame.origin.y = secondFrame.origin.y + cell.customLabel.frame.size.height + 10;
        cell.customLabel.frame = secondFrame;

        return cell;
}

and to calculate the height, in the tableView:heightForRowAtIndexPath: method:

- (CGFloat)tableView:(UITableView *)tableView heightForRowAtIndexPath:(NSIndexPath *)indexPath  {
	NSUInteger row = [indexPath row];
        NSString *element = [list objectAtIndex:row];
        CGFloat height = [element RAD_textHeightForSystemFontOfSize:17 xOffset:40.0] + 10.0;
        return height;
}

Tip 5 – UIFont: using the new fancy fonts but providing a callback if they are not available

iOS provides two chalk fonts, and I wanted to use any of those for the schoolboard: ChalkDuster and ChalkboardSE. However, these fonts are not available on all iOS devices.

Therefore, I check which ones are available in a FontUtil class and return accordingly:

+(UIFont*) getChalkFontWithSize:(int) size{
    if ([[UIFont fontNamesForFamilyName:@"ChalkDuster"] count] > 0){
        if ([[UIFont fontNamesForFamilyName:@"ChalkDuster"] containsObject:@"ChalkDuster"]){
            return [UIFont fontWithName:@"ChalkDuster" size:size];
        }
        else {
            return [[UIFont fontNamesForFamilyName:@"ChalkDuster"] objectAtIndex:0];
        }
    }
    else if ([[UIFont fontNamesForFamilyName:@"Chalkboard SE"] count] > 0){
        if ([[UIFont fontNamesForFamilyName:@"ChalkboardSE-Regular"] containsObject:@"ChalkboardSE-Regular"]){
            return [UIFont fontWithName:@"ChalkboardSE-Regular" size:size];
        }
        else {
            return [[UIFont fontNamesForFamilyName:@"Chalkboard SE"] objectAtIndex:0];
        }
    }
    else {
        return [UIFont systemFontOfSize:size];
    }
}

Another tip: to have a quick overview of all the iOS fonts, check www.iosfonts.com.

Tip 6 – UIColor: getting a color from an hex string

If you want to port an app from Android – we are integrating stuff here after all -, where colors are defined in their rgb values, this Stackoverflow Question gives a nice category on UIColor, which makes it possible to call:

[UIColor: colorWithHexString: @"#800000"]

returning a UIColor matching the hex rgb string.

Tip 7 – UIAlertView: recoloring the alert

To recolor the basic blue UIAlertView, you need to subclass it and override layoutSubViews. Again, I do not get why Apple did not build this feature into XCode, since it seems to me something a lot of developers want to do.

This blog post gives an excellent implementation of such a CustomAlertView.

You can then just set the color to #80000 or black to get a darkred or black color like this:

-(void) showRedAlert: (NSString *) message andTitle:(NSString *) title{
    [CustomAlertView setBackgroundColor:[UIColor colorWithHexString:@"800000"]
                        withStrokeColor:[UIColor whiteColor]];
	UIAlertView *alert = [[CustomAlertView alloc] initWithTitle:title
                                                        message:message delegate:nil cancelButtonTitle:@"Ok" otherButtonTitles:nil];
	[alert show];
	[alert release];
}

Resulting in alertviews looking like this:

This article is for people who played around in XCode and objectiveC already.

Tip 1: Learn to debug zombies

This section is useless to people using automatic reference counting(arc). However, there are still some reasons not to use arc, as discussed in this article on arc, and this section is aimed at the people that can not migrate just yet.

Especially when coming from a Java background where you are not used to release taken memory manually, you will make some allocation/releasing error sooner or later.

Note that when an instance is released, it becomes a zombie, and no method should be called on it anymore.

A common beginner mistake is to replace

[NSMutableArray new]

with something like

[NSMutableArray arrayWithArray: ...]

without modifying any other code for example.

When using new, the instance is not autoreleased; when using arrayWithArray, it is. You probably added a release statement companying the first statement. For the second statement you need to remove that release statement.

Unless the array is still needed later, with which the autorelease mechanism of the second statement might release it too early, requiring you to retain it, and still release it manually later.

Anyway, all of this is not too bad, but you have to learn how to track and solve these errors. By default, XCode will just make the simulator crash with a EXC_BAD_ACCESS, without giving any other information.

The first and most important thing to do is to press ALT-CMD-R in XCode, which will bring up the schemes screen:

Make sure you “Enable Zombie Objects” there and then run the app again. Reproduce the error. In the XCode output, a clear message will now be logged:

-[__NSArrayM release]: message sent to deallocated instance 0xc41d7e0

which is what we get if we just replace the [NSMutableArray new] with [NSMutableArray arrayWithArray:], without removing the release statement that comes somewhere after the [NSMutableArray new]. Since the second variation is autoreleased, the instance is already deallocated by the time the manual release statement on the array is called.

The above tells you the error type and on what object type and method the error occurred. But sometimes it is unclear why a certain instance was already deallocated. To bug these more challenging errors, you should profile the app through Product>Profile>Zombies.

When the fault occurs, the profiler blocks on the zombie object, and gives a stack of its entire lifecycle, which makes it easy to track these errors.

Tip 2: Code for both iPhone and iPad

When making an iOs app, make sure to have seperate screens for iPhone and iPad. The market for iPad apps is about as large as the one for iPhone apps, and no iPad user wants to use the default iPhone interface that puts an iPhone on his way bigger screen.

There are great tutorials out there to create a universal application. The one thing I want to add is that it is best to isolate all device dependent code in one Util class, such as:

#ifdef UI_USER_INTERFACE_IDIOM()
#define IS_IPAD() (UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPad)
#else
#define IS_IPAD() (false)
#endif

#import "UiUtil.h"

@implementation UiUtil

+(NSString *) getNibNameFrom: (NSString*) nibBaseName{
    if (IS_IPAD()){
        NSMutableString *nibForiPad = [NSMutableString stringWithString: nibBaseName];
        [nibForiPad appendString:@"-iPad"];
        return nibForiPad;
    }
    else {
         return nibBaseName;
    }
}

+(NSString *) getBackgroundNameFrom: (NSString*) backgroundBaseName{
    if (IS_IPAD()){
        NSMutableString *backgroundForiPad = [NSMutableString stringWithString: backgroundBaseName];
        [backgroundForiPad replaceOccurrencesOfString:@".png" withString:@"-iPad.png" options:0 range:NSMakeRange(0, [backgroundForiPad length])];
        return backgroundForiPad;
    }
    else {
        return backgroundBaseName;
    }
}

+(CGFloat) getFontModifier{
    if (IS_IPAD()){
        return 2;
    }
    else {
        return 1.0;
    }
}

@end

The above UiUtil class isolates the code to fetch the different xibs, backgrounds and fontsizes for iPhone or iPad, without putting any “isiPad” methods in your code.

Tip 3: Find your way to the Apple Developer examples

Recently, I had to use the GameKit and SystemConfiguration APIs and I found out that Apple is making awesome examples for all the iOs frameworks available to its developers in the iOS Developer library. The code in many of these examples can be reused for like 95%.

To integrate with the Game Center for example, you just need to download the GKTapper example and you can reuse the GameCenterManager class in it for all your projects.
All that you need to do to integrate with Game Center after importing this class is calling some submitScore and submitAchievement methods on it to send the data to the Apple servers, and instantiate GKAchievementViewController or GKLeaderboardViewController to show the achievements or leaderboard(s).

Writing the Game Center integration code can be done in an hour. I actually spend more time in entering the Game Center Achievements and Leaderboards in iTunes connect than I spend writing game center related code.

Tip 4: Use third party libs

Next to the awesome Apple developer examples, there are some great third party libs available as well, that can be used commercially.

One simple but great one is iRate. It enables you to ask a user for a rating on the appstore after certain conditions are met. It is easily configurable, and just consists of the iRate.h and iRate.m file.

You only need to put the following in your AppDelegate:

[iRate sharedInstance].appStoreID = 422387255;
//[iRate sharedInstance].debug = YES;

where appId is the App Id of your app as shown in Itunes Connect. By default, the dialog in which a rating is asked is not shown until at least 5 days have passed since the app was installed, but if debug is set to YES, the dialog shows right away on app startup.

1. Drupal Modules

A module is a collection of php functions that link into Drupal, providing additional functionality to the Drupal installation.
Because the module code executes within the Drupal context, it can access all the variables and structures and use all the functions of Drupal core.

1.1. Choose a short name

The first step in creating a module is to choose a short name for it. This short name will be used in all file and function names in your module. In this example, we will be using “nodes_to_text_file” as the short name.

Make sure your short name starts with a letter and only contains lower-case letters and underscores. Also make sure your module does not have the same short name as any theme you will be using.

1.2. Create the directory

In the modules directory of your Drupal 7 install, create a folder with your short name as its name.

1.3 empty module file

Create a nodes_to_text_file.module file in your nodes_to_text_file directory. It is to this file that we will add the functions that will hook into Drupal. In the next section, we will add the first function.
Module files begin with the opening php tag. Add it to the module file:

<?php

However, omit the closing php tag ?>. Omitting it is a convention, and including it might cause strange runtime behavior on certain server setups.

1.4 Creating the info file

Every module has its info file. It contains metadata about the module and hence, tells Drupal about the module. Without this file, the module will not appear in the module listing of your Drupal installation.

Create a nodes_to_text_file.info file in your nodes_to_text_file directory:

name = Nodes To Text File
description = A module that creates a text file containing all content of certain types of nodes whenever a node of one of these types is added, edited or deleted
core = 7.x

The above 3 properties form the minimal setup: name sets the name of the module, description describes what the module does and core determines with which version of Drupal this module is meant to be used.

After adding the info file, the module should be visible in your module listing. Go to Modules and scroll down to the “Other” section. The Nodes To Text File module should be listed there. Enable it.

2. Module Hooks

Hooks are the most important concept to grasp when implementing Drupal modules. Hooks provide the points at which you can insert your actions. They can be thought of as event listeners. An event such as deleting a node would trigger the hook “hook_node_delete”. In this case, Drupal will scan all modules for functions with the name mymodule_node_delete, where mymodule is the module’s name. If your module implements hook_node_delete, that function will always execute after a node gets deleted.

For a list of all available hooks in Drupal core, check the Drupal 7 hooks documentation.

2.1 Implementing the hook_help hook

The first hook we are going to implement is hook_help. This hook should always be implemented. It provides help and information about the module. To implement an hook, you need to replace “hook” in the hook name(f.e. hook_help) with your module’s short name and create a function in the .module file with that name.
In this case, nodes_to_text_file_help:

/**
 * Implements hook_help.
 *
 * Displays help and module information.
 *
 * @param path
 *   Which path of the site we're using to display help
 * @param arg
 *   Array that holds the current path as returned from arg() function
 */
function nodes_to_text_file_help($path, $arg) {
  switch ($path) {
    case "admin/help#nodes_to_text_file":
      return '<p>'.  t("A module that creates a text file containing all content of certain types of nodes whenever a node of one of these types is added, edited or deleted") .'</p>';
      break;
  }
}

The path parameter contains the information about where the user is currently at. If the user is currently at the help section of our module, we return a paragraph that gives information about our module. Note that we use the Drupal t($string) function, which translates the content if possible and which should be used on any string that is shown to the user.

Now, check Modules page of your Drupal site again. You should see a link ‘Help’ on the right of your module. If you click it, you will see the paragraph returned by the nodes_to_text_file_help function of our module.

2.2. Implementing the hooks hook_node_insert, hook_node_update and hook_node_delete

We will now implement the hooks hook_node_insert, hook_node_update and hook_node_delete.
All these hooks will call the same function nodes_to_text_file_write_file($node):

/**
 * Implements hook_node_delete.
 */
function nodes_to_text_file_node_delete($node){
	nodes_to_text_file_write_file($node);
}

/**
 * Implements hook_node_update.
 */
function nodes_to_text_file_node_update($node){
	nodes_to_text_file_write_file($node);
}

/**
 * Implements hook_node_insert.
 */
function nodes_to_text_file_node_insert($node){
	nodes_to_text_file_write_file($node);
}

For now, add the following implementation of the function:

function nodes_to_text_file_write_file($node){
	watchdog('We should write the file again!','Info Message');
}

watchdog is a useful function to debug hooks. They will be added to your Drupal server logging at Reports>Recent log messages:

<watchdog screenshot>

3. Database API

For this app, we will need two queries. One that fetches all the published nodes of the given content types of which the user is the owner:

/**
 * Custom content function.
 *
 * Retrieve data from database
 *
 * @return
 *   A result set of the targeted nodes.
 */
function nodes_to_text_file_contents($contentTypes){
  //Use Database db_select API to retrieve nodes.
  global $user;
  $uid = $user->uid;

  $queryResult = db_select('node', 'n')
    ->fields('n', array('nid', 'title', 'created'))
    ->condition('uid', $uid) //owner is current user.
    ->condition('status', 1) //Published.
    ->condition('type', $contentTypes, 'in') //Node type is in $contentTypes
    ->orderBy('created', 'DESC') //Most recent first.
    ->execute();
  return $queryResult;
}

This query uses the new(starting from Drupal 7) db_select function of the Database API. Other than the old db_query function, db_select makes it possible to construct the query in an object oriented matter.

db_select maps to a sql select statement(as do db_insert, db_update and db_delete map to their sql equivalents). It takes a table name(f.e. ‘node’) and an alias (f.e. ‘n’) as its arguments. This comes down to:

select * from node as n

fields selects the fields listed in the array in its second argument on the alias n:

select nid, title, created from node as n

condition adds a condition to the query. The first is the field, the second the value, the third the operator:

select nid, title, created from node as n where published = 1 and type in ('article','page') and uid = 1

if the values in the $contentTypes array are ‘article’ and ‘page’ and user Steffen has uid 1. Note that the “global $user” statement gets the global $user variable, which contains all information about the user that is currently logged in.

orderBy then sort the results according to the field in its first argument:

select nid, title, created from node as n where published = 1 and type in ('article','page') and uid = 1 order by created asc

The execute method then compiles and runs the query and returns the result.

The other query we need – one that fetches all content types of the core Node Module – uses the same API and is quite simular:

/**
 * Function that retrieves the content types from the database.
 *
 * Retrieve content types from database
 *
 * @return
 *   A result set of content types.
 */
function nodes_to_text_file_retrieve_content_types(){
  $queryResult = db_select('node_type', 'nt')
    ->fields('nt', array('type', 'name'))
    ->condition('module', 'node') //only node content types
    ->orderBy('type', 'ASC')
    ->execute();
  return $queryResult;
}

4. Drupal File API

We are now implementing our nodes_to_text_file_write_file function that is called by the node_insert, node_update and node_delete hooks. We briefly discuss the part in which we determine whether the file should be written and in which we fetch the contents first.

/**
 * Custom write file function.
 *
 * Creates the text file and stores it on the server.
 */
function nodes_to_text_file_write_file($node){
	$contentTypes = variable_get('nodes_to_text_file_content_types', array());
	$nodeType = $node->type;

	$elementFound = false;
	foreach ($contentTypes as &$value) {
    	if ($nodeType === $value){
    		$elementFound = true;
    		break;
    	}
	}

	if ($elementFound){
    	$queryResult = nodes_to_text_file_contents($contentTypes);

    	global $user;
    	$username = $user->name;

   	...
}

First, we fetch the node content types which should be included in the file using variable_get($name,$default) As we will see in the next section, variable_set($name,$value) – typically invoked in a module configuration form – sets Drupal system wide variables, which can then be fetched by variable_get.
We then match the node type of the node that was inserted, updated or deleted with the array of content types. If a match is found($elementFound), execution continues. We fetch the nodes using the query of the previous section and get a reference to the username of the currently logged in user.

Then – where the … dots are in the above function – we add the following and write out the fetched content to a file:

	if (! file_prepare_directory(file_build_uri($username))){
    		drupal_mkdir(file_build_uri($username));
    	}

    	$filename = file_build_uri($username . '/content.txt');
    	$fh = fopen($filename, 'w') or die("can't open file");

   	 	while($record = $queryResult->fetchAssoc()) {
    		$stringData = $record['nid'] . ' - ' . $record['title'] . ' - ' . $record['created'];
    		fwrite($fh, $stringData."\n");
    	} 

    	fclose($fh);

We write the fetched contents to a context.txt file in the default Drupal public files location, in a folder with the username as its name. If this folder does not exist yet, we create it first:

if (! file_prepare_directory(file_build_uri($username))){
   drupal_mkdir(file_build_uri($username));
}

Then we construct the filename:

$filename = file_build_uri($username . '/content.txt');

If the username is “Steffen”, then the filename will be:
<drupal installation location>/sites/default/files/steffen/content.txt

Then we open the file for writing and iterate over the queryResult. We print out the nid, the title and the created timestamp of every record to a new line in the file. We then close the file again:

   $fh = fopen($filename, 'w') or die("can't open file");

   while($record = $queryResult->fetchAssoc()) {
      $stringData = $record['nid'] . ' - ' . $record['title'] . ' - ' . $record['created'];
      fwrite($fh, $stringData."\n");
   } 

   fclose($fh);

We are using some functions of the Drupal File API here:
file_build_uri: This function constructs, given a relative path, a URI into Drupal’s default files location(this is usually something like ‘sites/default/files/’).
file_prepare_directory: This function checks that the directory exists and is writable.
drupal_mkdir: This function creates a directory using Drupal’s default mode.

To respectively open the file for writing, write to it and close it again, we use the php functions fopen, fwrite and fclose.

5. Drupal Form API

Our module is implemented now. However, we want to be able to configure the node content types for which the file should be written. We already saw that we were fetching this array of content types like this:

$contentTypes = variable_get('nodes_to_text_file_content_types', array());

What remains is to explain how we can set this array of content types first through a module configuration form.

We will implement hook_menu so we get a configuration form at our module at the relative path admin/config/content/nodes_to_text_file:

/**
 * Implements hook_menu().
 */
function nodes_to_text_file_menu() {
  $items = array();  

  $items['admin/config/content/nodes_to_text_file'] = array(
    'title' => 'Nodes To Text File',
    'description' => 'Configuration for Nodes To Text File module',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('nodes_to_text_file_form'),
    'access arguments' => array('access administration pages'),
    'type' => MENU_NORMAL_ITEM
  );

  return $items;
}

Using the hook_menu hook, modules register paths to define how URL requests are handled. In this case, we are setting a path that makes a configuration form available from the Drupal Configuration page:

We passed the form already to the array of the ‘page arguments’ property of the link, but we didnt define it yet. We are doing so now:

/**
 * Form function, called by drupal_get_form()
 * in nodes_to_text_file_menu().
 */
function nodes_to_text_file_form($form, &$form_state) {
	$queryResult = nodes_to_text_file_retrieve_content_types();

	$options = array();
	while($record = $queryResult->fetchAssoc()) {
    	$options[$record['type']] = t($record['name']);
    } 

  	$form['nodes_to_text_file_content_types'] = array(
    	'#type' => 'checkboxes',
    	'#title' => t('Content types to include'),
    	'#default_value' => variable_get('nodes_to_text_file_content_types', array()),
    	'#description' => t('The content types to include.'),
    	'#options' => $options
  	);

  return system_settings_form($form);
}

First we are creating the array of possible content types so we can use it later to generate the necessary checkboxes on the form:

$queryResult = nodes_to_text_file_retrieve_content_types();

	$options = array();
	while($record = $queryResult->fetchAssoc()) {
    	$options[$record['type']] = t($record['name']);
    }

In Drupal, forms should be constructed using the Form API. We are using it to construct our form:

$form['nodes_to_text_file_content_types'] = array(
    	'#type' => 'checkboxes',
    	'#title' => t('Content types to include'),
    	'#default_value' => variable_get('nodes_to_text_file_content_types', array()),
    	'#description' => t('The content types to include.'),
    	'#options' => $options
  	);

after which we return the form. Note that the Form API is pretty straightforward. It is generally enough to consult the API page to know what types of fields are available and which properties are available and which ones are required.

6. The entire module file for reference

<?php
/**
 * Implements hook_help.
 *
 * Displays help and module information.
 *
 * @param path
 *   Which path of the site we're using to display help
 * @param arg
 *   Array that holds the current path as returned from arg() function
 */
function nodes_to_text_file_help($path, $arg) {
  switch ($path) {
    case "admin/help#nodes_to_text_file":
      return '<p>'.  t("A module that creates a text file containing all content of certain types of nodes whenever a node of one of these types is added, edited or deleted") .'</p>';
      break;
  }
}

/**
 * Implements hook_node_delete.
 */
function nodes_to_text_file_node_delete($node){
	nodes_to_text_file_write_file($node);
}

/**
 * Implements hook_node_update.
 */
function nodes_to_text_file_node_update($node){
	nodes_to_text_file_write_file($node);
}

/**
 * Implements hook_node_insert.
 */
function nodes_to_text_file_node_insert($node){
	nodes_to_text_file_write_file($node);
}

/**
 * Custom write file function.
 *
 * Creates the text file and stores it on the server.
 */
function nodes_to_text_file_write_file($node){
	$contentTypes = variable_get('nodes_to_text_file_content_types', array());
	$nodeType = $node->type;

	$elementFound = false;
	foreach ($contentTypes as &$value) {
    	if ($nodeType === $value){
    		$elementFound = true;
    		break;
    	}
	}

	if ($elementFound){
    	$queryResult = nodes_to_text_file_contents($contentTypes);

    	global $user;
    	$username = $user->name;

   	 	if (! file_prepare_directory(file_build_uri($username))){
    		drupal_mkdir(file_build_uri($username));
    	}

    	$filename = file_build_uri($username . '/content.txt');
    	$fh = fopen($filename, 'w') or die("can't open file");

   	 	while($record = $queryResult->fetchAssoc()) {
    		$stringData = $record['nid'] . ' - ' . $record['title'] . ' - ' . $record['created'];
    		fwrite($fh, $stringData."\n");
    	} 

    	fclose($fh);
	}
}

/**
 * Custom content function.
 *
 * Retrieve data from database
 *
 * @return
 *   A result set of the targeted nodes.
 */
function nodes_to_text_file_contents($contentTypes){
  //Use Database db_select API to retrieve nodes.
  global $user;
  $uid = $user->uid;

  $queryResult = db_select('node', 'n')
    ->fields('n', array('nid', 'title', 'created'))
    ->condition('uid', $uid) //owner is current user.
    ->condition('status', 1) //Published.
    ->condition('type', $contentTypes, 'in') //Node type is in $contentTypes
    ->orderBy('created', 'DESC') //Most recent first.
    ->execute();
  return $queryResult;
}

/**
 * Function that retrieves the content types from the database.
 *
 * Retrieve content types from database
 *
 * @return
 *   A result set of content types.
 */
function nodes_to_text_file_retrieve_content_types(){
  $queryResult = db_select('node_type', 'nt')
    ->fields('nt', array('type', 'name'))
    ->condition('module', 'node') //only node content types
    ->orderBy('type', 'ASC')
    ->execute();
  return $queryResult;
}

/**
 * Form function, called by drupal_get_form()
 * in nodes_to_text_file_menu().
 */
function nodes_to_text_file_form($form, &$form_state) {
	$queryResult = nodes_to_text_file_retrieve_content_types();

	$options = array();
	while($record = $queryResult->fetchAssoc()) {
    	$options[$record['type']] = t($record['name']);
    } 

  	$form['nodes_to_text_file_content_types'] = array(
    	'#type' => 'checkboxes',
    	'#title' => t('Content types to include'),
    	'#default_value' => variable_get('nodes_to_text_file_content_types', array()),
    	'#description' => t('The content types to include.'),
    	'#options' => $options
  	);

  return system_settings_form($form);
}

/**
 * Implements hook_menu().
 */
function nodes_to_text_file_menu() {
  $items = array();  

  $items['admin/config/content/nodes_to_text_file'] = array(
    'title' => 'Nodes To Text File',
    'description' => 'Configuration for Nodes To Text File module',
    'page callback' => 'drupal_get_form',
    'page arguments' => array('nodes_to_text_file_form'),
    'access arguments' => array('access administration pages'),
    'type' => MENU_NORMAL_ITEM
  );

  return $items;
}

Making a backup of a Mysql database and restoring it

Making a backup of a MySql database is straightforward. I am giving the statements below for a Mac, but on Windows the same mysqldump and mysql commands can be used from the bin folder of the MySql installation.

First navigate to your MySql installation folder. This will probably be something like:

cd /usr/local/mysql

And then run mysqldump:

sudo ./bin/mysqldump -u <username> -p <password> testdatabase > /development/testdatabase.sql

where <username> is your mysql username and <password> is your mysql password.

This will generate a sql script in the development folder that is able to recreate all tables and indexes and reinsert all data that are available in the testdatabase, by running the following command on the target server:

sudo ./bin/mysql testdatabase < /development/testdatabase.sql

assuming the testdatabase.sql was first sent to the development folder on the target server by using the scp command or some graphical client.

However, when running the above mysqldump command on Mac or Windows, all tablenames are stored in lowercase in the target sql script file by default. When it is then imported on a Linux server, all tablenames are in lowercase as well.

This is a problem, since the case of the tablenames matters on Linux machines. It is possible that applications – written in PHP, Java or something else – that will make use of the database expect some or all of these tablenames to be in upper case or have its first letter capitalized. These applications will then not run correctly.

Case sensitivity: the problem when migrating a MySql database from Windows/Mac to Linux

Hence, it is necessary the cases are preserved in the generated sql script. The answer to this Stackoverflow question suggests to restart the mysql database on the Mac/windows server with the lower_case_table_names parameter set to 0. For more information about this, please check the Mysql documentation about the lower_case_table_names parameter.

However, you might not have the permissions to take the database offline, restart it with this parameter and then run mysqldump again. You are then stuck with the script with only lower case table names. I think the only option is then to edit the script and replace all lower case occurences with the version with the proper case.

The method that I describe below is just a quick way to do this using Java.

Getting the list of tablenames with their case right

Most applications – such as Java web applications (using JPA/Hibernate for example), and Drupal and WordPress installations – are able to automatically create the database tables they need if they are missing. Hence, you can run the application once and make sure the tables are generated. You then open a MySql terminal.

On the Linux machine, you would run the following from the mysql installation bin folder:

mysql --user=<user> --password=<password>

After which you will get the mysql terminal.
Assuming the tables were already generated by the web application in the testdatabase database, you run:

mysql> show tables from testdatabase;

Which in my case renders:

+--------------------------------+
| Tables_in_testdatabase         |
+--------------------------------+
| Account_                       |
| Address                        |
| AnnouncementsDelivery          |
| AnnouncementsEntry             |
| AnnouncementsFlag              |
| AssetCategory                  |
| AssetCategoryProperty          |
| AssetEntries_AssetCategories   |
| AssetEntries_AssetTags         |
| AssetEntry                     |
| AssetLink                      |
| AssetTag                       |
| AssetTagProperty               |
| AssetTagStats                  |
| AssetVocabulary                |
| BlogsEntry                     |
| BlogsStatsUser                 |
| BookmarksEntry                 |
| BookmarksFolder                |
| BrowserTracker                 |
| CalEvent                       |
| Chat_Entry                     |
| Chat_Status                    |
| ClassName_                     |
| ClusterGroup                   |
| Company                        |
| Contact_                       |
| Counter                        |
| Country                        |
| CyrusUser                      |
| CyrusVirtual                   |
| DLFileEntry                    |
| DLFileRank                     |
| DLFileShortcut                 |
| DLFileVersion                  |
| DLFolder                       |
| EmailAddress                   |
| ExpandoColumn                  |
| ExpandoRow                     |
| ExpandoTable                   |
| ExpandoValue                   |
| Group_                         |
| Groups_Orgs                    |
| Groups_Permissions             |
| Groups_Roles                   |
| Groups_UserGroups              |
| IGFolder                       |
| IGImage                        |
| Image                          |
| JournalArticle                 |
| JournalArticleImage            |
| JournalArticleResource         |
| JournalContentSearch           |
| JournalFeed                    |
| JournalStructure               |
| JournalTemplate                |
| Layout                         |
| LayoutPrototype                |
| LayoutSet                      |
| LayoutSetPrototype             |
| ListType                       |
| Lock_                          |
| MBBan                          |
| MBCategory                     |
| MBDiscussion                   |
| MBMailingList                  |
| MBMessage                      |
| MBMessageFlag                  |
| MBStatsUser                    |
| MBThread                       |
| Mail_Account                   |
| Mail_Attachment                |
| Mail_Folder                    |
| Mail_Message                   |
| MembershipRequest              |
| OpenSocial_Gadget              |
| OrgGroupPermission             |
| OrgGroupRole                   |
| OrgLabor                       |
| Organization_                  |
| PasswordPolicy                 |
| PasswordPolicyRel              |
| PasswordTracker                |
| Permission_                    |
| Phone                          |
| PluginSetting                  |
| PollsChoice                    |
| PollsQuestion                  |
| PollsVote                      |
| Portlet                        |
| PortletItem                    |
| PortletPreferences             |
| QUARTZ_BLOB_TRIGGERS           |
| QUARTZ_CALENDARS               |
| QUARTZ_CRON_TRIGGERS           |
| QUARTZ_FIRED_TRIGGERS          |
| QUARTZ_JOB_DETAILS             |
| QUARTZ_JOB_LISTENERS           |
| QUARTZ_LOCKS                   |
| QUARTZ_PAUSED_TRIGGER_GRPS     |
| QUARTZ_SCHEDULER_STATE         |
| QUARTZ_SIMPLE_TRIGGERS         |
| QUARTZ_TRIGGERS                |
| QUARTZ_TRIGGER_LISTENERS       |
| RatingsEntry                   |
| RatingsStats                   |
| Region                         |
| Release_                       |
| ResourceAction                 |
| ResourceCode                   |
| ResourcePermission             |
| Resource_                      |
| Role_                          |
| Roles_Permissions              |
| SCFrameworkVersi_SCProductVers |
| SCFrameworkVersion             |
| SCLicense                      |
| SCLicenses_SCProductEntries    |
| SCProductEntry                 |
| SCProductScreenshot            |
| SCProductVersion               |
| SN_MeetupsEntry                |
| SN_MeetupsRegistration         |
| SN_WallEntry                   |
| ServiceComponent               |
| Shard                          |
| ShoppingCart                   |
| ShoppingCategory               |
| ShoppingCoupon                 |
| ShoppingItem                   |
| ShoppingItemField              |
| ShoppingItemPrice              |
| ShoppingOrder                  |
| ShoppingOrderItem              |
| SocialActivity                 |
| SocialEquityAssetEntry         |
| SocialEquityGroupSetting       |
| SocialEquityHistory            |
| SocialEquityLog                |
| SocialEquitySetting            |
| SocialEquityUser               |
| SocialRelation                 |
| SocialRequest                  |
| Subscription                   |
| TasksProposal                  |
| TasksReview                    |
| Team                           |
| Ticket                         |
| UserGroup                      |
| UserGroupGroupRole             |
| UserGroupRole                  |
| UserIdMapper                   |
| UserTracker                    |
| UserTrackerPath                |
| User_                          |
| Users_Groups                   |
| Users_Orgs                     |
| Users_Permissions              |
| Users_Roles                    |
| Users_Teams                    |
| Users_UserGroups               |
| Vocabulary                     |
| WSRP_WSRPConsumer              |
| WSRP_WSRPConsumerPortlet       |
| WSRP_WSRPProducer              |
| WebDAVProps                    |
| Website                        |
| WikiNode                       |
| WikiPage                       |
| WikiPageResource               |
| WorkflowDefinitionLink         |
| WorkflowInstanceLink           |
| blocked_user                   |
| deleted_message                |
| private_message                |
| read_message                   |
+--------------------------------+
176 rows in set (0.55 sec)

The above is a list of tablenames for the Liferay 6.0 portal and migrating these tables is a case in which I actually used the method described here. I stripped the list of all pipes(|) and saved the list of tablenames into a txt file, one tablename per line.

If you cannot generate the list of tablenames with the right case automatically like above, you will need to manually create this file, severely slowing up the method used in this tutorial.

Setting the case right with some Java code

The below Java class takes the testdatabase.sql file and the tablenames.txt file as input and writes the sql script with the cases set right to a testdatabase-converted.sql file:

public class SetCaseRight {

	private static List<String> readFile(String fileName){
		try{
			List<String> lines = new ArrayList<String>();
			FileInputStream fstream = new FileInputStream(fileName);
			DataInputStream in = new DataInputStream(fstream);
			BufferedReader br = new BufferedReader(new InputStreamReader(in));
			String strLine;
			while ((strLine = br.readLine()) != null)   {
				lines.add(strLine);
			}
			in.close();
			return lines;
		}catch (Exception e){
			e.printStackTrace();
			return null;
		}
	}

	private static String readFileAsString(String fileName){
        try {
			StringBuffer fileData = new StringBuffer(1000);
			BufferedReader reader = new BufferedReader(new FileReader(fileName));
			char[] buf = new char[1024];
			int numRead=0;
			while((numRead=reader.read(buf)) != -1){
			    String readData = String.valueOf(buf, 0, numRead);
			    fileData.append(readData);
			    buf = new char[1024];
			}
			reader.close();
			return fileData.toString();
		} catch (Exception e) {
			e.printStackTrace();
			return null;
		}
    }

	private static void writeStringToFile(String s, String fileName){
		try {
			BufferedWriter out = new BufferedWriter(new FileWriter(fileName));
			out.write(s);
			out.close();
		}
		catch (IOException e) {
			e.printStackTrace();
		}
	}

	public static void main(String[] args){
		String sqlSourceFile;
		String tableNamesTxtFile;
		String sqlTargetFile;
		if (args.length >= 1){ sqlSourceFile = args[0];} else { sqlSourceFile = "testdatabase.sql";}
		if (args.length >= 2){ tableNamesTxtFile = args[1];} else { tableNamesTxtFile = "tablenames.txt";}
		if (args.length >= 3){ sqlTargetFile = args[2];} else { sqlTargetFile = "testdatabase-converted.sql";}
		String sql = getSql(sqlSourceFile);
		for (String tableName: readTableNames(tableNamesTxtFile)){
			sql = sql.replaceAll("`(?i)" + tableName.trim()+"`", "`"+tableName.trim()+"`");
		}
		writeStringToFile(sql, sqlTargetFile);
	}

	public static String getSql(String sqlSourceFile){
		return readFileAsString(sqlSourceFile);
	}

	public static List<String> readTableNames(String tableNamesTxtFile){
		return readFile(tableNamesTxtFile);
	}
}

It is also possible to run the above class command line – without having to compile it yourself – by using this SetCaseRight class file:

java SetCaseRight testdatabase.sql tablenames.txt testdatabase-converted.sql

in a folder where java is on the PATH and where the first argument is the sqlTargetFile, the second argument is the tableNamesTxtFile and the third argument is the sqlTargetFile.

The resulting sql file is then ready to be imported on the Linux server.

Setting up the theme project

We are assuming you will be using Eclipse, but will always tell what is going on under the hood, so other IDE(or commandline!) guys should also be able to follow.

You should start out by downloading the Liferay Plugins SDK from the bottom of the Liferay additional files page.
Eclipse users should also install the Liferay IDE Eclipse Plugin and add a Liferay Server(and runtime) after. More info on how to do this can be found in the Liferay documentation on installing the Liferay IDE or at this very own blog, in the last part of the getting started with portals and Spring Portlet MVC post.

If you were already working with Liferay, you probably had these installed already anyway and the only thing you need to do then, is start your Liferay server.

Then, within Eclipse, choose New>Project>Liferay>Liferay Project and then check Theme in the box.

What the theme plugin does is run a script from the <liferay-sdk-home>/themes folder that generates a Liferay theme project for you, in your Eclipse workspace.
You can run the script manually by entering

./create.sh integrating-stuff "Integrating Stuff"

or

create.bat integrating-stuff "Integrating Stuff"

in a terminal from the said <sdkhome>/themes folder.

The Liferay IDE also takes a look at your Liferay runtime and copies a bunch of files to the docroot folder of your project.

These files represent the _styled theme, a very basic Liferay theme, which can be found in the portal source in the portal-web/docroot/html/themes folder or in the docroot/html/themes folder of your Liferay install.
As the Liferay documentation indicates, almost all themes will use the “_styled” theme as a base. If you want to start from scratch, the parent of your theme would be the “_unstyled” theme. Inheriting from “_unstyled” would give you quite a bit more work but would give you full control of the theme.

Any of the files that were copied to your docroot folder can be overridden by your theme by placing them in your _diffs folder. In other words, the _diffs folder will contain all files that differ from your parent theme.

Note that, as already indicated above, the “_styled” theme is not production ready. Since our _diffs folder is still empty, our theme – which is currently not overriding anything and is an exact representation of the “_styled” theme – would not look that great if we would deploy it to Liferay right now.
Therefore, it is a better idea to copy the content of the _diffs folder of the production ready theme you want to start from into your own theme’s _diffs folder. To start from the default “Classic” Liferay theme, download the portal source(link) and copy the _diffs folder from the Classic theme at portal-web/docroot/html/themes to your own theme’s _diffs folder.
Now your theme is production ready and ready for deployment.

2. Deploying the theme to Liferay

You probably already noticed the ant build script in the root of your project. If you run the default ant goal “deploy” of this script, the theme is published to your Liferay server.
(If you are not doing this from within Eclipse, you need to properly set the values in the build.properties file in the ssdk’s theme folder.)

Let’s deploy and enable our theme on our Liferay server, which for now, looks exactly the same as the default “Classic” theme. If the theme deployed successfully, the Liferay logging should contain something like:

3. Overriding parts of the parent theme

The content of the_diffs folder we copied from the Classic theme consisted of 3 folders: css, images and js(javascript). In the _styled theme there is one additional folder, templates, of which the classic theme does not override anything. This is recommended practice: themes should try to avoid overriding template files since they are likely to change in between Liferay versions so changing them might make porting the theme to a new Liferay version more difficult.

a. overriding css

Now, to add a nice gradient background image and change the default text size and font, we have to modify our custom.css file.

We will change the

body {
	background: #EEF0F2;
	font-size: 11px;
}

into:

body {
        background-color: #4F555B;
	background-image: url("../images/layout/bg-grad.gif");
	background-repeat: repeat-x;
	color: #FFFFFF;
	font-family: Verdana,Helvetica,sans-serif;
	font-size: 11px;
}

and also create a layout_folder in our images folder and add the file bg-grad.gif to it.

If we redeploy the theme and load our Liferay page, we see that the background has changed and text appears a bit different. Pretty easy, huh.

b. Overriding and adding images

In the previous section, we already added a background image to the theme. If you want to override an image, you just need to add it to the images folder as well, but the image needs to have the same folder structure and file name as the one you want to override.
For example, if you want to override some emoticons, you would create an emoticons folder(you need to look at the _styled theme files to know what you can override) and start putting names there of files that also exist in the parent theme(angry.gif, blank.gif,..).

c. Overriding the custom javascript

This can be done the same way images are overridden. However, the _styled theme only has one javascript file: main.js, which, with Liferay 6, only contains some logic that keeps the Liferay breadcrumbs always on screen.

d. Overriding the template files

As mentioned already, the Liferay documentation advises to avoid overriding the template files. That being said, almost all projects I know off involved creating themes that were modifying these templates.
If one wants to add or remove parts of the page and the modifications go beyond styling dom elements, these templates are usually changed. The view technology these files use is Velocity.

The two most important ones are portal-normal.vm – which renders the whole page – and portlet.vm – which renders one portlet. The other ones respectively define additional velocity variables(init_custom.vm), render the navigation bar(navigation.vm) and render the popup pages(portal_pop_up.vm).

Let’s take a look at the portal-normal.vm Velocity template:

<!DOCTYPE html>

#parse ($init)

<html class="#language("lang.dir")" dir="#language("lang.dir")" lang="$w3c_language_id">

<head>
	<title>$the_title - $company_name</title>

	$theme.include($top_head_include)
</head>

<body class="$css_class">

#if($is_signed_in)
	#dockbar()
#end

<div id="wrapper">
	<a href="#main-content" id="skip-to-content">#language("skip-to-content")</a>

	<header id="banner" role="banner">
		<hgroup id="heading">
			<h1 class="company-title">
				<a class="logo" href="$company_url" title="#language("go-to") $company_name">
					<span>$company_name</span>
				</a>
			</h1>

			<h2 class="community-title">
				<a href="$community_default_url" title="#language("go-to") $community_name">
					<span>$community_name</span>
				</a>
			</h2>

			<h3 class="page-title">
				<span>$the_title</span>
			</h3>
		</hgroup>

		#if(!$is_signed_in)
			<a href="$sign_in_url" id="sign-in" rel="nofollow">$sign_in_text</a>
		#end

		#if ($update_available_url)
			<div class="popup-alert-notice">
				<a class="update-available" href="$update_available_url">#language("updates-are-available-for-liferay")</a>
			</div>
		#end

		#if ($has_navigation)
			#parse ("$full_templates_path/navigation.vm")
		#end
	</header>

	<div id="content">
		<nav class="site-breadcrumbs" id="breadcrumbs">
			<h1>
				<span>#language("breadcrumbs")</span>
			</h1>

			#breadcrumbs()
		</nav>

		#if ($selectable)
			$theme.include($content_include)
		#else
			$portletDisplay.recycle()

			$portletDisplay.setTitle($the_title)

			$theme.wrapPortlet("portlet.vm", $content_include)
		#end
	</div>

	<footer id="footer" role="contentinfo">
		<p class="powered-by">
			#language("powered-by") <a href="http://www.liferay.com" rel="external">Liferay</a>
		</p>
	</footer>
</div>

</body>

$theme.include($bottom_include)

</html>

If one would want (and is allowed to) remove the “Powered by Liferay” line, he can easily do this by just removing the line containing the text.

If the static htmldesign/prototypes are already made by web designers, it is easy for developers to port this created design to Liferay by copying the html to the portal-normal.vm file and inserting the Velocity directives where required.
All these Velocity templates are clearly readable. When taking a look at the above file, it is obvious how to replace the whole Liferay header, for example.

4. More advanced stuff

There is more advanced stuff one can use to fine tune a theme and is not covered by this tutorial – such as adding settings or color schemes to a theme. Although this tutorial gets you started from a practical point of view, it is probably a good idea to scan through the Liferay documentation on themes as well.

WebServer class

Everything that we are going to use is already present in the Android API. We are not going to need any additional libs.

A pragmatic implementation looks like this:

package com.integratingstuff.android.webserver;

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketException;

import org.apache.http.HttpException;
import org.apache.http.impl.DefaultConnectionReuseStrategy;
import org.apache.http.impl.DefaultHttpResponseFactory;
import org.apache.http.impl.DefaultHttpServerConnection;
import org.apache.http.params.BasicHttpParams;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.BasicHttpProcessor;
import org.apache.http.protocol.HttpRequestHandlerRegistry;
import org.apache.http.protocol.HttpService;
import org.apache.http.protocol.ResponseConnControl;
import org.apache.http.protocol.ResponseContent;
import org.apache.http.protocol.ResponseDate;
import org.apache.http.protocol.ResponseServer;

import android.content.Context;

public class WebServer {

	public static boolean RUNNING = false;
	public static int serverPort = 8080;

	private static final String ALL_PATTERN = "*";
	private static final String EXCEL_PATTERN = "/*.xls";
	private static final String HOME_PATTERN = "/home.html";

	private Context context = null;

	private BasicHttpProcessor httpproc = null;
	private BasicHttpContext httpContext = null;
	private HttpService httpService = null;
	private HttpRequestHandlerRegistry registry = null;

	public WebServer(Context context) {
		this.setContext(context);

		httpproc = new BasicHttpProcessor();
		httpContext = new BasicHttpContext();

		httpproc.addInterceptor(new ResponseDate());
		httpproc.addInterceptor(new ResponseServer());
		httpproc.addInterceptor(new ResponseContent());
		httpproc.addInterceptor(new ResponseConnControl());

		httpService = new HttpService(httpproc,
		    new DefaultConnectionReuseStrategy(), new DefaultHttpResponseFactory());

		registry = new HttpRequestHandlerRegistry();

		//registry.register(HOME_PATTERN, new HomeCommandHandler(context));

		httpService.setHandlerResolver(registry);
	}

	private ServerSocket serverSocket;

	public void runServer() {
		try {
			serverSocket = new ServerSocket(serverPort);

			serverSocket.setReuseAddress(true);

			while (RUNNING) {
				try {
					final Socket socket = serverSocket.accept();

					DefaultHttpServerConnection serverConnection = new DefaultHttpServerConnection();

					serverConnection.bind(socket, new BasicHttpParams());

					httpService.handleRequest(serverConnection, httpContext);

					serverConnection.shutdown();
				} catch (IOException e) {
					e.printStackTrace();
				} catch (HttpException e) {
					e.printStackTrace();
				}
			}

			serverSocket.close();
		} catch (SocketException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}

		RUNNING = false;
	}

	public synchronized void startServer() {
		RUNNING = true;
		runServer();
	}

	public synchronized void stopServer() {
		RUNNING = false;
		if (serverSocket != null) {
			try {
				serverSocket.close();
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}

	public void setContext(Context context) {
		this.context = context;
	}

	public Context getContext() {
		return context;
	}
}

In the WebServer contructor we are initializing an HttpService. We are using all the default implementations here. The interesting part is the construction of the HttpRequestHandlerRegistry. As we will see, we will add a custom requesthandler to this registry which shall build the response for a particular request.

In the runServer method we actually start the server by initiating a low level ServerSocket and make it accept connections. We then continiously bind the socket to our webserver implementation and tell it to handle the request.

The startServer and stopServer methods are the methods actually used to start and stop the server.

In Android apps, a server like this should be run as an Android Service. So, we are implementing the following WebServerService:

package com.integratingstuff.android.webserver;

import android.app.Service;
import android.content.Intent;
import android.os.IBinder;
import android.util.Log;

public class WebServerService extends Service {

	private WebServer server = null;

	@Override
	public void onCreate() {
		Log.i("HTTPSERVICE", "Creating and starting httpService");
		super.onCreate();
		server = new WebServer(this);
		server.startServer();
	}

	@Override
	public void onDestroy() {
		Log.i("HTTPSERVICE", "Destroying httpService");
		server.stopServer();
		super.onDestroy();
	}

	@Override
	public IBinder onBind(Intent intent) {
		return null;
	}
}

Which we can start from the Android application like this:

Intent webServerService = new Intent(context, WebServerService.class);
context.startService(webServerService);

HttpRequestHandler

Our WebServer is now running, but we did not register any requesthandlers yet. In our webserver, we commented the following line:

registry.register(HOME_PATTERN, new HomeCommandHandler(context));

Let’s uncomment it and implement the HomeCommandHandler:

package com.integratingstuff.android.webserver;

import java.io.IOException;
import java.io.OutputStream;
import java.io.OutputStreamWriter;

import org.apache.http.HttpEntity;
import org.apache.http.HttpException;
import org.apache.http.HttpRequest;
import org.apache.http.HttpResponse;
import org.apache.http.entity.ContentProducer;
import org.apache.http.entity.EntityTemplate;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpRequestHandler;

import android.content.Context;

public class HomeCommandHandler implements HttpRequestHandler {
	private Context context = null;

	public HomeCommandHandler(Context context) {
		this.context = context;
	}

	@Override
	public void handle(HttpRequest request, HttpResponse response,
	    HttpContext httpContext) throws HttpException, IOException {
		HttpEntity entity = new EntityTemplate(new ContentProducer() {
			public void writeTo(final OutputStream outstream) throws IOException {
				OutputStreamWriter writer = new OutputStreamWriter(outstream, "UTF-8");
				String resp = "<html><head></head><body><h1>Home<h1><p>This is the homepage.</p></body></html>";

				writer.write(resp);
				writer.flush();
			}
		});
		response.setHeader("Content-Type", "text/html");
		response.setEntity(entity);
	}

	public Context getContext() {
		return context;
	}
}

We are mapping the pattern home.html to this requesthandler. This handler will send some basic html back as its response.

If you now surf to http://<ip-of-device>:8080/home.html, you should see the homepage with the “This is the homepage” message.

So now, our webserver is up and running and responding to home.html requests. You can easily add more requesthandlers to the registry, that match specific patterns or wildcard * patterns.

MultipartParser

The above can be easily found elsewhere, so to give this article some added value, I will quickly indicate below how to parse form date/get the contents of files contained in the request.

Code

We will need to parse a multipart request manually. I am just going to give the code snippets and I am not going into the parsing intrinsics, but basically, the http protocol defines boundary delimiters for multipart messages. What we are doing with the following MultipartParser is read the entire sent form into a Multivalued map(a map of which the value is a list of a certain type of object, so every key is mapped to a list of 1 or more values). Note that the MultipartParser actually uses a CaseInsensitiveMultivaluedMap that wraps a regular MultivaluedMap, because keys like “name” and “NAME” map to the same formproperty.

MultipartParser

package com.integratingstuff.android.webserver;

import java.io.IOException;
import java.io.InputStream;

public class MultipartParser {

	public final static String SEP = "\n";

	private InputStream is;
	private byte[] boundaryBA;
	static private byte[] boundaryDelimiterBA = "--".getBytes();

	private MultivaluedMap partHeaders;
	private PartInputStream partIS;

	private static int BOUNDARY_TYPE_START = 0;
	private static int BOUNDARY_TYPE_END = 1;
	// private static int BOUNDARY_TYPE_INVALID = 2;

	private byte[] buff;
	// the next byte to return
	private int buffIdx = 0;
	// The number of bytes that were set on the buffer (red from is);
	private int buffSize = 0;
	// the position of the next boundary ("--boundary"), -1 if does not exist in
	// this buffer
	private int boundryIdx = -1;
	// The index of the byte that is suspected of been the boundary
	private int saveIdx = 0;

	// This is a temp array that is used to read stuff into it.
	private byte[] temp = new byte[1024];

	public MultipartParser(InputStream is, String boundary) {
		this.is = is;
		boundaryBA = ("--" + boundary).getBytes();
		// make sure to allocate a buffer that is at least double then the
		// boundary length
		int buffLength = Math.max(8192, boundaryBA.length * 2);
		buff = new byte[buffLength];

	}

	private void shiftBuff() {
		System.arraycopy(buff, buffIdx, buff, 0, buffSize - buffIdx);

		buffSize -= buffIdx;
		saveIdx -= buffIdx;
		if (saveIdx < 0)
			saveIdx = 0;
		/*
		 * throw new RuntimeException("This should never happend, we found a bug.");
		 */
		boundryIdx = Math.max(-1, boundryIdx - buffIdx);
		buffIdx = 0;

		// for debug purposes
		// Arrays.fill(buff, buffIdx, buff.length-1, (byte)0);

	}

	public boolean nextPart() throws IOException {
		// if this is the first next just get rid of the PREAMBLE
		if (partIS == null) {
			partIS = new PartInputStream();
		}
		// clear the part/preamble bytes that were not read
		digestPartStream();
		if (digestBoundary() == BOUNDARY_TYPE_END)
			return false;
		partIS.setState(PartInputStream.STATE_NOT_ACTIVE);
		partIS = new PartInputStream();
		partHeaders = parseHeaders();
		return partHeaders != null;
	}

	public InputStream getPartBodyStream() {
		return partIS;
	}

	public MultivaluedMap getPartHeaders() {
		return partHeaders;
	}

	// read till end of stream (next boundary)
	private void digestPartStream() throws IOException {
		while (partIS.read(temp) != -1) {
		}
	}

	private boolean compareByte(byte[] a, int aOffset, byte[] b, int bOffset,
	    int length) {
		for (int i = 0; i < length; i++) {
			if (a[aOffset + i] != b[bOffset + i])
				return false;
		}
		return true;
	}

	private int digestBoundary() throws IOException {
		// it might be that there is a new line before the boundary
		digestNewLine();

		// promote pointers to the end of the boundary
		buffIdx += boundaryBA.length;
		saveIdx += boundaryBA.length; // DO NOT DELETE

		// check if this is an end boundary
		int unredBytes = verifyByteReadyForRead(2);
		if (unredBytes >= 2) {
			if (compareByte(buff, buffIdx, boundaryDelimiterBA, 0,
			    boundaryDelimiterBA.length))
				return BOUNDARY_TYPE_END;
		}
		// OK
		digestNewLine();
		boundryIdx = -1;
		findBounderyIfNeeded();
		return BOUNDARY_TYPE_START;
	}

	private void findBounderyIfNeeded() {
		if (boundryIdx == -1) {
			boundryIdx = indexOf(buff, saveIdx, buffSize, boundaryBA);
			if (boundryIdx != -1) {
				int nlSize = 0;
				if (boundryIdx > 1) {
					if (buff[boundryIdx - 2] == '\r' && buff[boundryIdx - 1] == '\n')
						nlSize = 2;
					else
						nlSize = 1;
				}
				if (boundryIdx == 1) {
					nlSize = 1;
				}

				saveIdx = boundryIdx - nlSize;
			} else {
				// the boundary was not found, but we can promote the save till
				// boundary size + NL size
				saveIdx = Math.max(saveIdx, buffSize - (boundaryBA.length + 2));
			}
		}
	}

	private int verifyByteReadyForRead(int required) throws IOException {
		int unreadBytes = buffSize - buffIdx - 1;
		if (unreadBytes < required) {
			fetch(required - unreadBytes);
			unreadBytes = buffSize - buffIdx;
		}
		return unreadBytes;
	}

	private int fetch(int minmum) throws IOException {
		int res = 0;
		int max2featch = buff.length - buffSize;

		if (max2featch < minmum) {
			shiftBuff();
			max2featch = buff.length - buffSize;
		}

		while (res < minmum && max2featch > 0) {
			max2featch = buff.length - buffSize;

			int read = is.read(buff, buffSize, max2featch);
			if (read == -1) {
				if (res == 0)
					return -1;
				else
					break;
			}
			res += read;
			buffSize += read;
		}
		findBounderyIfNeeded();
		return res;

	}

	private void digestNewLine() throws IOException {
		// make sure we have enough byte to read
		int unreadBytes = verifyByteReadyForRead(2);
		int size = 0;

		if (unreadBytes >= 2 && buff[buffIdx] == '\r' && buff[buffIdx + 1] == '\n')
			size = 2;
		else if (buff[buffIdx] == '\r')
			size = 1;
		else if (buff[buffIdx] == '\n')
			size = 1;
		buffIdx += size;
		if (saveIdx < buffIdx)
			saveIdx = buffIdx;
	}

	private int indexOf(byte[] ba, int start, int end, byte[] what) {
		for (int i = start; i < end - what.length + 1; i++) {
			// only if the first byte equals do the compare (to improve
			// performance)
			if (ba[i] == what[0])
				if (compareByte(ba, i, what, 0, what.length))
					return i;
		}
		return -1;
	}

	private MultivaluedMap parseHeaders() throws IOException {

		MultivaluedMap headers = new CaseInsensitiveMultivaluedMap();

		String line;
		do {
			line = readLine();
			if (line == null || line.equals(""))
				break;
			int semIdx = line.indexOf(":");
			headers.add(line.substring(0, semIdx).trim(), line.substring(semIdx + 1)
			    .trim());

		} while (true);
		if (saveIdx < buffIdx)
			saveIdx = buffIdx;
		return headers;
	}

	private String readLine() throws IOException {

		int lineIdx = 0;
		int breakeSize = 0;
		while (lineIdx <= verifyByteReadyForRead(lineIdx)) {
			if (buff[buffIdx + lineIdx] == '\n') {
				breakeSize = 1;
				break;
			}
			if (buff[buffIdx + lineIdx] == '\r') {
				if ((verifyByteReadyForRead(lineIdx + 1) >= lineIdx + 1)
				    && (buff[buffIdx + lineIdx + 1] == '\n')) {
					breakeSize = 2;
					break;
				} else {
					breakeSize = 1;
					break;
				}
			}
			lineIdx++;
		}

		// got to the end of input without NL
		if (lineIdx == 0) {
			buffIdx += breakeSize;
			return null;
		}

		String hdr = new String(buff, buffIdx, lineIdx);
		buffIdx += lineIdx + breakeSize;
		return hdr;
	}

	public class PartInputStream extends InputStream {
		// The state of the part Stream
		// 0 active
		// 1 not active (the Parser already moved to the next part.)
		private int state = 0;
		public final static int STATE_ACTIVE = 0;
		public final static int STATE_NOT_ACTIVE = 1;

		public void setState(int status) {
			this.state = status;

		}

		@Override
		public int read(byte[] b, int off, int len) throws IOException {
			if (state == STATE_NOT_ACTIVE) {
				throw new IOException(
				    "Stream allready closed: the PartInputStream is not accesable after moving to the next part");
			}
			int available = verifyNumOfByteToReadB4Boundary(len);
			if (available < 1) {
				return available
                        }
			int size2copy = Math.min(len, available);
			System.arraycopy(buff, buffIdx, b, off, size2copy);
			buffIdx += size2copy;
			return size2copy;
		}

		@Override
		public int read() throws IOException {
			if (state == STATE_NOT_ACTIVE) {
				throw new IOException(
				    "Stream allready closed: the PartInputStream is not accesable after moving to the next part");
			}
			int i = verifyNumOfByteToReadB4Boundary(1);
			if (i < 1)
				return -1;
			// make sure that the return value is 0 - 255
			int res = buff[buffIdx] & 0xff;
			if (res < 0) {
				int t = 0;
				t++;
			}
			buffIdx++;
			return res;

		}

		private int verifyNumOfByteToReadB4Boundary(int minmum) throws IOException {
			int availabe = saveIdx - buffIdx;
			if (availabe >= minmum)
				return availabe;
			//
			if (saveIdx <= boundryIdx) {
				if (availabe == 0)
					return -1;
				return availabe;
			}
			int fetched = fetch(minmum - availabe);
			availabe = saveIdx - buffIdx;
			if (availabe == 0 && fetched == -1)
				return -1;

			return availabe;

		}

		@Override
		public int available() {
			return saveIdx - buffIdx;
		}
	}
}