Java, why?

Просто два факта.

1) У объектов java.net.URL методы hashCode и equals делают DNS Resolve хостнейма указанного в адресе и (sic!) сравнивают два адреса используя их айпи адреса.
Таким образом,

Из-за DNS-балансировщика new URL("http://google.com") != new URL("http://google.com")  //в разное время,

Два сайта на одном айпи new URL("http://example.com") == new URL("http://example.net") 

И самое смешное. В HashMap однажды добавленное значение по ключу URL(google.com) может быть никогда не получено, так как хешкод меняется. Особенно заметно когда URL сериализуется и отправляется на другой узел, тогда никакой dns-кеш на любом уровне не сработает и все весело упадет.



2) Enum.hashCode() в джаве возвращает адрес в памяти. Те же приколы с сериализацией или даже запуском одного и того же кода в разных джава-машинах.

Делая шардинг мапы на java, выбор ноды для хранения/поиска данных по хеш-коду, например.. веселого дебага.

Caching Maven Local Repository in Docker

I'm using Docker instances as Jenkins slaves and run my containers like

# docker run -d --name="Chewbakka" antigluk/jenkins-slave-centos7-java -labels docker-centos7-java -name "Chewbakka"

(actually, I do it using docker-jenkins-slave )

However, it doesn't make sense to keep maven local repository in every container.

It can be done using Docker Data Volumes
We can mount host's directory into any directory inside container if we specify -v argument like this:

-v /tmp/docker-m2cache:/root/.m2:rw

This will mount host's directory /tmp/docker-m2cache into container's /root/.m2

Resulting command will be

# docker run -d --name="Chewbakka" -v /tmp/docker-m2cache:/root/.m2:rw antigluk/jenkins-slave-centos7-java -labels docker-centos7-java -name "Chewbakka"

Debug Beeline Client for Hive

Just a note how to enable debug mode in beeline (or any other) Hive client.

To enable remote debugging, we need to pass "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005" arguments to JVM.

Tricky part is to find the place where JVM is being executed. It's file $HADOOP_HOME/hive-client/bin/ext/beeline.sh - in Hortonworks (HDP) installation it will be /usr/hdp/current/hive-client/bin/ext/beeline.sh

On the line
exec $HADOOP jar ${beelineJarPath} $CLASS $HIVE_OPTS "$@"

but -Xdebug option should be placed to HADOOP_CLIENT_OPTS variable:

export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j.configuration=beeline-log4j.properties -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"
 

-Xdebug - enables remote debugging
-Xrunjdwp - sets configuration properties, where
server - start server or try to connect to debugger (usually server=y to do remote debug via Idea)
suspend - freeze start of program and wait until debugger connects to it
address -  port of server you will connect to

Now if you just start # beeline
you'll see that debug server started on port 5005, and you can connect to it via Idea (or whatever).

Protected method in Java

The protected modifier specifies that the member can only be accessed within its own package (sic! - just in case) (as with package-private) and, in addition, by a subclass of its class in another package.

http://docs.oracle.com/javase/tutorial/java/javaOO/accesscontrol.html

Reinventing the wheel: "ORM" with PropertiesConfiguration

"ORM" with PropertiesConfiguration

It was interesting to try creating simple database (I needed to store different objects/tables in single key-value storage, that was temporary and quite reasonable for that situation), based on 'properties' file.

Dependencies
1) Useful gson library to convert beans to/from json (serialization)
https://code.google.com/p/google-gson/

2) PropertiesConfiguration class is just like Properties but supports auto saving and reloading from file (persistence)
http://commons.apache.org/proper/commons-configuration/apidocs/org/apache/commons/configuration/PropertiesConfiguration.html


Building ORM:

1. Let's create our key-value storage:


public class PersistentConfiguration extends PropertiesConfiguration {
    public PersistentConfiguration(String fileName) throws ConfigurationException {
        super();

        File config = new File(fileName);
        setFile(config);
        this.setAutoSave(true);
        this.setReloadingStrategy(new FileChangedReloadingStrategy());
        this.setDelimiterParsingDisabled(true);
        this.setListDelimiter((char) 0);

        if (config.exists()) {
            this.load();
        }
    }
}

It's file based storage. Since we need to store json, need to make it not treat commas as list separators.

2.

To store objects we need every object to have ID, and store last created index to avoid collisions after deleting some elements.

Let's use "<class name>:index" key name for last index, and
"<class name>:<id>" for table rows (see getIndexPropertyName and getItemPropertyName in Storage)

Our beans should implement this interface, to be sure we can access ID of row:
public interface Indexed {
    String getId();
    void setId(String id);
}

And finally, the storage:

public class Storage {
    protected final Gson gson = new Gson();
    private PersistentConfiguration config = null;

    public PigStorage() {
        try {
            config = new PersistentConfiguration("./data.properties");
        } catch (ConfigurationException e) {
            e.printStackTrace();
        }
    }

    public synchronized void store(Indexed obj) {
        String modelIndexingPropName = getIndexPropertyName(obj.getClass());

        if (obj.getId() == null) {
            int lastIndex = config.getInt(modelIndexingPropName, 0);
            lastIndex ++;
            config.setProperty(modelIndexingPropName, lastIndex);
            obj.setId(Integer.toString(lastIndex));
        }

        String modelPropName = getItemPropertyName(obj.getClass(), Integer.parseInt(obj.getId()));
        String json = gson.toJson(obj);
        config.setProperty(modelPropName, json);
    }

    public synchronized Indexed load(Class model, int id) throws ItemNotFound {
        String modelPropName = getItemPropertyName(model, id);
        if (config.containsKey(modelPropName)) {
            String json = config.getString(modelPropName);
            return (Indexed) gson.fromJson(json, model);
        } else {
            throw new ItemNotFound();
        }
    }

    public synchronized void delete(Class model, int id) {
        String modelPropName = getItemPropertyName(model, id);
        config.clearProperty(modelPropName);
    }

    public boolean exists(Class model, int id) {
        return config.containsKey(getItemPropertyName(model, id));
    }


    private String getIndexPropertyName(Class model) {
        return String.format("%s:index", model.getName());
    }

    private String getItemPropertyName(Class model, int id) {
        return String.format("%s.%d", model.getName(), id);
    }
}

Done!

Improvements:
making SELECT with filtering

    public synchronized List loadAll(Class model, FilteringStrategy filter) {
        ArrayList<Indexed> list = new ArrayList<Indexed>();
        String modelIndexingPropName = getIndexPropertyName(model);
        LOG.info(String.format("Loading all %s-s", model.getName()));
        int lastIndex = getConfig().getInt(modelIndexingPropName, 0);
        for(int i=1; i<=lastIndex; i++) {
            try {
                Indexed item = load(model, i);
                if ((filter == null) || filter.is_conform(item)) {
                    list.add(item);
                }
            } catch (ItemNotFound ignored) {
            }
        }
        return list;
    }

Where FilteringStrategy is:

public interface FilteringStrategy {
    boolean is_conform(Indexed item);
}

For example:
public class OnlyOwnersFilteringStrategy implements FilteringStrategy {
    private final String username;

    public OnlyOwnersFilteringStrategy(String username) {
        this.username = username;
    }

    @Override
    public boolean is_conform(Indexed item) {
        Owned object = (Owned) item;
        return object.getOwner().compareTo(username) == 0;
    }
}