Off Heap Cache - OHCache - Simple Example

Reference

What

OHC 是一個 off heap 的 cache library

Why

如果是用 map 做 cache, 資料都在 heap 裡面, heap 裡面的問題就是
  1. 會被 GC, GC 就會週期性的對 JVM 效能產生影響
  2. 如果要 persist 需要另外實作
如果是放在 off heap, 比方說用 mmap, 或是用 MappedByteBuffer access 的記憶體, 就是放在 heap memory 之外, 由作業系統控制著. 
好處是記憶體量不用被 heap size 限制, 缺點是要自己小心控制, 不然會造成系統問題.'

How

Maven

<dependency>
 <groupId>org.caffinitas.ohc</groupId>
 <artifactId>ohc-core</artifactId>
 <version>0.4.4</version>
</dependency>

Codes

public class SimpleMain {

    public static void main(String[] params) {
        Scanner scanner = new Scanner(System.in);
        while(scanner.hasNextLine()) {
            if (scanner.nextLine().trim().equals("GO")) {
                System.out.println(Runtime.getRuntime().freeMemory()/1024/1024 + "/" + Runtime.getRuntime().totalMemory()/1024/1024);
                OHCache<String,String> c = OHCacheBuilder.<String,String>newBuilder()
                        .keySerializer(new StringSerializer())
                        .valueSerializer(new StringSerializer())
                        .build();
                c.put("A", "A");
                System.out.println(c.get("A"));
                IntStream.range(0, 1000_000).forEach(idx -> {
                    c.put(UUID.randomUUID().toString(), UUID.randomUUID().toString());
                });
                System.out.println(Runtime.getRuntime().freeMemory()/1024/1024 + "/" + Runtime.getRuntime().totalMemory()/1024/1024);
            } else {
                System.out.println("input 'GO' then click enter");
            }
        }
    }

    private static class StringSerializer implements CacheSerializer<String> {

        @Override
        public void serialize(String s, ByteBuffer byteBuffer) {
            byteBuffer.put(s.getBytes());
        }

        @Override
        public String deserialize(ByteBuffer byteBuffer) {
            return StandardCharsets.UTF_8.decode(byteBuffer).toString();
        }

        @Override
        public int serializedSize(String s) {
            return s.getBytes().length;
        }
    }

}

Output


這張是輸入了好幾次 GO, 每次都會輸入一百萬次 key value 到 cache.

jconsole 與系統資源使用圖可以看到 heap 只用不到 100MB, 但實際上已經用了 600MB.
就是都放在 off heap 中

ForkJoinPool - Thread Management

Reference

Question

原本以為 new ForkJoinPool(2) 像這樣的宣告, 是讓 ForkJoinPool 最多維持 2 個 thread 來執行.
可是當實際觀察的時候, 卻發現有非常多個 thread 被叫起來跑.
比方說原本 fork -> join 的範例, 如果把 thread name 印出來, 就會發現有很多個 thread 被叫起來.
而且每秒印出的 pool 也會發現 pool size 變大.

Codes: Thread size exceed parallel level

public class ForkJoinPoolManyThreadMain {

    static final CountDownLatch latch = new CountDownLatch(1);
    private static final AtomicInteger threadCount = new AtomicInteger();

    public static void main(String[] params) throws ExecutionException, InterruptedException {
        ForkJoinPool pool = new ForkJoinPool(2);
        ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
        s.scheduleAtFixedRate(() -> {
            System.out.println(pool); }, 0, 1, TimeUnit.SECONDS);
        pool.execute(new MainAction());
        latch.await();
        s.shutdown();
        pool.shutdown();
    }


    private static class MainAction extends RecursiveAction {

        @Override
        protected void compute() {
            List<SubAction> actions = IntStream.range(0,100).mapToObj(idx -> new SubAction()).collect(toList());
            actions.forEach(SubAction::fork);
            actions.forEach(SubAction::join);
            latch.countDown();
        }
    }

    private static class SubAction extends RecursiveAction {

        @Override
        protected void compute() {
            try {
                System.out.println(Thread.currentThread().getName() + " sleep 1 seconds");
                TimeUnit.SECONDS.sleep(1);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }


}

Output

從印出來的 log 就可以發現: thread 超過 2 個到達 14 個. pool size 實際上也到達 16





Requirement

發現這件事情的時候覺得無法理解, 同時也發現原本以為 main task 的 thread 是跟被 join 的 thread 合再一起了. 結果看起來是有很多新的 thread 被產生出來執行. 如此我們怎麼確保不會有過多的 thread 被建立出來? 

Suggestion: Customized ThreadFactory? (Don't do this)

因為 ForkJoinPool 裡的這段程式, 網路上有人建議是提供自己的 thread factory, 如果超過指定的 thread 數量就回傳 null. 如此就被 ForkJoinPool 當成一個錯誤而繼續使用既有的 thread.

這個做法的確可以控制建立的 thread 數量 , 不過回傳 null 對 ForkJoinPool 來說是一個錯誤, 以 ForkJoinPool 的規則來說這個時候是要 createWorker 的, 不應該出現錯誤而回傳 null.



How: Join after done

突然想到會產生新的 thread 是不是因為 join 導致 MainAction 停住, 但又不是正在執行, 所以為了避免 starvation 使 ForkJoinPool 需要建立一個新的 thread. 所以試著先判斷 isDone, done 才 join. 結果就可以了.  (但還不清楚原因)

Working Code

public class ForkJoinPoolFixedThreadMain {

    public static final CountDownLatch latch = new CountDownLatch(1);

    public static void main(String[] params) throws ExecutionException, InterruptedException {
        ForkJoinPool pool = new ForkJoinPool();
        MainAction action = new MainAction();
        ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
        s.scheduleAtFixedRate(() -> System.out.println(pool), 0, 1, TimeUnit.SECONDS);
        pool.execute(action);
        latch.await();
        s.shutdown();
    }

    private static class MainAction extends RecursiveAction {

        @Override
        protected void compute() {
            List<SubAction> actions = IntStream.range(0,100).mapToObj(idx -> new SubAction()).collect(toList());
            actions.forEach(SubAction::fork);
            while(!actions.isEmpty()) {
                for (Iterator<SubAction> i = actions.iterator(); i.hasNext(); ) {
                    SubAction action = i.next();
                    if (action.isDone()) {
                        action.join();
                        i.remove();
                    }
                }
            }
            latch.countDown();
        }
    }

    private static class SubAction extends RecursiveAction {

        @Override
        protected void compute() {
            long start = System.currentTimeMillis();
            while (true) {
                if (System.currentTimeMillis() - start > 1000) {
                    break;
                }
            }
        }
    }


}

Output

因為呼叫 new ForkJoinPool() 所以 pool size 最多就維持在 8 (我的電腦有 8 core)

ForkJoinPool - Error Handling

Reference

Error Handling

如果 fork 出去的 task/action 有 exception, 呼叫 join 的時候要注意 catch exception.
不然 task/action 就死掉失控了.
而且可怕的是: 不會有 log.

Codes

這段程式是說每個 action 有個 id, 奇數 id 會報 exception.

Exception without catch when join

先是不要 catch 的情況
public class ForkJoinPoolErrorMain {

    static final CountDownLatch latch = new CountDownLatch(1);

    public static void main(String[] params) throws InterruptedException {
        ForkJoinPool pool = new ForkJoinPool(2);
        ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
        s.scheduleAtFixedRate(() -> System.out.println(pool), 0, 1, TimeUnit.SECONDS);
        pool.submit(new ErrorAction(MAIN_TASK_ID));
        latch.await(10, TimeUnit.SECONDS);
        pool.shutdown();
        s.shutdown();
    }

    static class ErrorAction extends RecursiveAction {

        public static final int MAIN_TASK_ID = -1;
        private final int id;

        ErrorAction(int id) {
            this.id = id;
        }

        private boolean isSubTask() {
            return id >= 0;
        }

        @Override
        protected void compute() {
            if (isSubTask()) {
                if (id % 2 == 1) {
                    throw new IllegalStateException("Error when id is odd number");
                }
            } else {
                List<ErrorAction> actions = IntStream.range(0,10).mapToObj(idx -> new ErrorAction(idx)).collect(Collectors.toList());
                actions.forEach(ErrorAction::fork);
                actions.forEach(action -> {
                    action.join();
                });
                latch.countDown();
            }
            System.out.println("compute done. id=" + id);
        }
    }

}
輸出如下, 不過一個 exception 也沒有.
main action 也沒有完成.(main action id = -1)











Exception with catch statement

有 catch 的程式如下
public class ForkJoinPoolErrorMain {

    static final CountDownLatch latch = new CountDownLatch(1);

    public static void main(String[] params) throws InterruptedException {
        ForkJoinPool pool = new ForkJoinPool(2);
        ScheduledExecutorService s = Executors.newScheduledThreadPool(1);
        s.scheduleAtFixedRate(() -> System.out.println(pool), 0, 1, TimeUnit.SECONDS);
        pool.submit(new ErrorAction(MAIN_TASK_ID));
        latch.await(10, TimeUnit.SECONDS);
        pool.shutdown();
        s.shutdown();
    }

    static class ErrorAction extends RecursiveAction {

        public static final int MAIN_TASK_ID = -1;
        private final int id;

        ErrorAction(int id) {
            this.id = id;
        }

        private boolean isSubTask() {
            return id >= 0;
        }

        @Override
        protected void compute() {
            if (isSubTask()) {
                if (id % 2 == 1) {
                    throw new IllegalStateException("Error when id is odd number");
                }
            } else {
                List<ErrorAction> actions = IntStream.range(0,10).mapToObj(idx -> new ErrorAction(idx)).collect(Collectors.toList());
                actions.forEach(ErrorAction::fork);
                actions.forEach(action -> {
                    try {
                        action.join();
                    } catch (Exception ex) {
                        System.err.println(ex + ", id=" + action.id);
                    }
                });
                latch.countDown();
            }
            System.out.println("compute done. id=" + id);
        }
    }

}

這個程式就可以印 error, 而且 maintask 可以執行完成











Exception with ExecutorService

如果是 ExecutorService, 就算是丟 RuntimeException.
ExecutorService 還是會把 exception 印到 console 上.
public class ExecutorServiceWithExceptionMain {

    public static void main(String[] params) {
        ExecutorService ex = Executors.newSingleThreadExecutor();
        ex.execute(() -> {
            throw new IllegalStateException("test throw illegalState");
        });
        ex.shutdown();
    }


}
即使是直接丟 exception 也會印到 console.

ForkJoinPool - Work stealing


Reference

前篇: Java Concurrency - ForkJoinPool 的 deadlock
下篇: ForkJoinPool - Error Handling

Introduction

ForkJoinPool 實作 work stealing 的概念, 簡單說明一下

Work-stealing

  1. 每個 thread 會有一個 queue, queue 裡面放的是 CPU bound 的工作
  2. queue 裡面的工作可能會產生新的能平行作業的工作
  3. 新產生的工作會放在 queue 裡面
  4. 如果有 thread 把 queue 裡面的工作做完了, 就會去別的 queue 拿工作來處理 (steal)

Java Concurrency - ForkJoinPool 的 deadlock

Reference

Will ForkJoinTask encounter deadlock problem?

在看 ForkJoinTask 的時候, 看到 ForkJoinTask 可以先 fork 去執行, 再來呼叫 join 等待結束.
如圖, 些動作總共需要幾個 thread?
原本想:
1. MainTask 一個 thread, 兩個 subtask 各一個 thread 執行.
2. 當兩個 subtask 要 join 的時候, MainTask 會等待
如此一來, 若 ForkJoinPool 的 thread 只有一個, MainTask 佔一個, 那 subtask 要 fork 再 join 不就沒有 thread 可以處理?

測試之後發現, MainTask 的 thread 其實會在呼叫 join 的時候就被交出去.
因此當只有一個 thread, MainTask 的 thread 會執行到呼叫 subtask.join 的時候就離開, 讓出 thread 去執行 subtask 的任務.
舉例來說
public class ForkJoinSingleThreadMain {

    private static final CountDownLatch latch = new CountDownLatch(1);

    public static void main(String[] params) throws InterruptedException {
        ForkJoinPool pool = new ForkJoinPool(1);
        pool.submit(new MainTask());
        latch.await();
    }

    private static class MainTask extends RecursiveAction {

        @Override
        protected void compute() {
            Subtask subtask1 = new Subtask(1);
            Subtask subtask2 = new Subtask(2);
            System.out.println(Thread.currentThread().getName() + " fork subtask1");
            subtask1.fork();
            System.out.println(Thread.currentThread().getName() + " fork subtask2");
            subtask2.fork();
            System.out.println(Thread.currentThread().getName() + " join subtask1");
            subtask1.join();
            System.out.println(Thread.currentThread().getName() + " join subtask2");
            subtask2.join();
            System.out.println(Thread.currentThread().getName() + " join done");
            latch.countDown();
        }
    }

    private static class Subtask extends RecursiveAction {

        private final int id;

        Subtask(int id) {
            this.id = id;
        }

        @Override
        protected void compute() {
            System.out.println(Thread.currentThread().getName() + ":" + id + " compute");
            try {
                TimeUnit.MILLISECONDS.sleep(300);
            } catch (InterruptedException e) {
            }
            System.out.println(Thread.currentThread().getName() + ":" + id + " compute done");
        }
    }


}
這個 class 的執行結果是:
可以看到當 MainTask 呼叫 join 的時候就把 thread 交出去了.

那會 deadlock 嗎?
其實就是看 subtask 是否要搶 lock, 如果有就可能會 deadlock.
下面這個例子就可以製造 deadlock.
不過 ForkJoinPool 不能宣告為 new ForkJoinPool(1) 因為這樣只會有一個 thread 執行
至少要 new ForkJoinPool(2) 以上, 才可以有兩個 thread 搶 lock 造成 deadlock

public class ForkJoinPoolDeadlockMain {

    private static final CountDownLatch l = new CountDownLatch(1);
    private static final Object lock1 = new Object();
    private static final Object lock2 = new Object();

    public static void main(String[] params) throws InterruptedException {
        ForkJoinPool pool = new ForkJoinPool(2);
        pool.submit(ForkJoinTask.adapt(new Runnable(){
            @Override
            public void run() {
                Obj1Locker locker1 = new Obj1Locker();
                Obj2Locker locker2 = new Obj2Locker();
                locker1.fork();
                locker2.fork();
                locker1.join();
                locker2.join();
                l.countDown();
            }
        }));
        l.await();
    }

    private static class Obj1Locker extends RecursiveAction {

        @Override
        public void compute() {
            synchronized (lock1) {
                System.out.println("obj1Locker got lock1");
                sleep();
                synchronized (lock2) {
                    System.out.println("obj1Locker got lock2");
                }
            }
        }


    }

    private static class Obj2Locker extends RecursiveAction {

        @Override
        public void compute() {
            synchronized (lock2) {
                System.out.println("obj2Locker got lock2");
                sleep();
                synchronized (lock1) {
                    System.out.println("obj2Locker got lock1");
                }
            }
        }


    }

    private static void sleep() {
        try {
            TimeUnit.SECONDS.sleep(1);
        } catch (InterruptedException e) {
        }
    }

}

Java Concurrent - ForkJoinPool

ExecutorService and Problem

用一個 ExecutorService submit task 後, 如果 task 會 submit 其他的 task, 這些 task 都會放進 ExecutorService 的 queue 中.
這樣就沒辦法做到 "第一個 task 與其產生的 subtask 完成後就印一行 log"

ForkJoinPool and Purpose

使用了 ForkJoinPool 就可以透過 task 的 fork 與 join 讓 task 變成樹狀結構.
  1. 建立 ForkJoinPool
  2. 用 ForkJoinPool submit task
  3. task 建立幾個新的 subtask
  4. 呼叫 subtask 的 fork
  5. 呼叫 subtask 的 join
  6. 執行到 parent 的 task 結束
如此就能讓 task 自然分群

Reference and Sample

Code

public class ForkJoinPoolMain {

    private static AtomicInteger actionCounter = new AtomicInteger();
    public static void main(String[] params) throws ExecutionException, InterruptedException {
        ForkJoinPool pool = new ForkJoinPool();
        System.out.println("main:" + pool.submit(new MyAction(new int[]{1,2,3,4,5,6,7,8,9,10,11})).get());
    }

    private static class MyAction extends RecursiveTask<Integer> {
        // Extends RecursiveAction if no return value required

        private int count;
        private int[] numbers;

        public MyAction(int[] numbers) {
            this.count = actionCounter.incrementAndGet();
            this.numbers = numbers;
        }

        @Override
        protected Integer compute() {
            if (numbers.length > 2) {
                System.out.println(count + " split " + Arrays.toString(numbers) + " to 2 arrays");
                List<MyAction> actions = new ArrayList<>();
                actions.add(new MyAction(Arrays.copyOfRange(numbers, 0, numbers.length / 2)));
                actions.add(new MyAction(Arrays.copyOfRange(numbers, numbers.length / 2, numbers.length)));
                actions.forEach(MyAction::fork);
                return actions.stream().mapToInt(MyAction::join).sum();
            } else if (numbers.length == 2) {
                System.out.println(count + " add " + numbers[0] + " and " + numbers[1]);
                return numbers[0] + numbers[1];
            }
            System.out.println(count + " just return " + numbers[0]);
            return numbers[0];
        }
    }

}
下篇:ForkJoinPool 沒處理好也會 deadlock

Java Concurrent CyclicBarrier

Introduction

CyclicBarrier 很久沒用都忘了, 來複習一下..

Description

簡單說就是很多個 thread 去作事情, 作完之後就 await.
CyclicBarrier 等作完的 thread 達到指定的個數後, await 就結束.

Code

public class CyclicBarrierMain {

    public static void main(String[] params) throws InterruptedException {
        int runner = 5;
        CountDownLatch gameOver = new CountDownLatch(runner);
        CyclicBarrier b = new CyclicBarrier(runner, () -> {
            System.out.println("barrier done");
        });
        IntStream.range(0,runner).forEach(idx -> {
            new Thread() {
                @Override
                public void run() {
                    try {
                        System.out.println("wait " + idx);
                        b.await();
                        System.out.println("count down " + idx);
                        gameOver.countDown();
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }
            }.start();
        });
        gameOver.await();
        System.out.println("game over");
    }

}

Java Concurrent Phaser

Introduction

今天才知道 Java 7 有提供一個 Phaser class 來處理更複雜的 CountDownLatch 需求.

Simple Phaser

想用 Phaser, 首先你的 threads 工作要有分階段, 一個階段沒做完就不能進行下個階段.
有了這個限制後就可以照下面的流程做事情
  1. 先註冊, 看有幾個 thread 要做事情.
    => 透過呼叫 Phaser#register 去註冊
  2. 做完之後呼叫 arrive, 告訴 Phaser 有事情做完了
  3. 當每個 thread 都把事情做完, Phaser 就會進入下一個階段, 稱做 advance
  4. 當進入下一階段, Phaser 的 onAdvance method 就會被呼叫,
    這個 method 是留給我們 developer 去實做
以上. 簡單的概念, 覺得看好久都沒看懂, 希望看這篇的人可以簡單看懂.
接著是範例, 這範例是

  1. 假設有幾個人賽跑
  2. 首先要到達起跑線 
  3. 都到了以後才開跑 
  4. 每個人都到達終點舊比賽結束.

public class PhaserRacingGame {

    public static void main(String[] params) {
        Random r = new Random();
        Phaser perGamePhase = new Phaser() {
            @Override
            protected boolean onAdvance(int phase, int registeredParties) {
                System.out.println("phase:" + phase + ", registeredParties:" + registeredParties + ", arrived:" + getArrivedParties());
                switch (phase) {
                    case 0:
                        System.out.println("start!");
                        break;
                    case 1:
                        System.out.println("game finish");
                        break;
                    default:
                        System.out.println("unexpected:" + phase);
                }
                return super.onAdvance(phase, registeredParties);
            }
        };
        IntStream.range(0,5).forEach(idx -> {
            perGamePhase.register();
            new Thread(() -> {
                System.out.println(idx + " arrive starting line and wait for start");
                perGamePhase.arriveAndAwaitAdvance();
                System.out.println(idx + " run!! startTime:" + System.currentTimeMillis());
                long spendTime = r.nextInt(1000);
                sleep(spendTime);
                perGamePhase.arrive();
            }).start();
        });
        sleep(5000);
    }

    private static void sleep(long ms) {
        try {
            TimeUnit.MILLISECONDS.sleep(ms);
        } catch (InterruptedException e) {
        }
    }

}

Parent Phaser

再來比較複雜的是 Parent Phaser, Parent Phaser 的目的是減少 synchronization 的時候搶 lock 的碰撞. 
使用 Parent Phaser 的時候要注意的是: advance 是發生在 Parent Phaser 而不是 Child Phaser, 
使用 Parent Phaser 的時候, 是可以確保每個 Child Phaser 準備號都進入下一個 phase 的時候, 已經先到達的 Child Phaser 才可以更進一步 arrive.
比方說世足十六強賽, 要等所有能進入八強賽的隊伍都出線之後才可以開始進行八強賽.
那 Parent Phaser 就是賽程, 十六強賽則是有八個 Child Phaser.
以下範例是看出特性, 方便 copy paste 觀察
public class ParentPhaser {

    public static void main(String[] params) {
        Phaser parent = new Phaser() {
            @Override
            protected boolean onAdvance(int phase, int registeredParties) {
                System.out.println("parent phase:" + phase + ", registeredParties:" + registeredParties);
                return super.onAdvance(phase, registeredParties);
            }
        };
        Phaser child1 = new Phaser(parent, 4) {
            @Override
            protected boolean onAdvance(int phase, int registeredParties) {
                System.out.println("child phase:" + phase + ", registeredParties:" + registeredParties);
                return super.onAdvance(phase, registeredParties);
            }
        };
        Phaser child2 = new Phaser(parent, 3) {
            @Override
            protected boolean onAdvance(int phase, int registeredParties) {
                System.out.println("child phase:" + phase + ", registeredParties:" + registeredParties);
                return super.onAdvance(phase, registeredParties);
            }
        };
        IntStream.range(0,10).forEach(idx -> {
            child1.arrive();
            child1.arrive();
            child1.arrive();
            child1.arrive();
//            child1.arrive(); // cause error, child1 need wait for child2 finish
            child2.arrive();
            child2.arrive();
            child2.arrive();
        });
    }


}

Zip files

In a project, I need zip file and append to separated folders.

public static void main(String[] params) throws IOException {
 List<String> srcFiles = Arrays.asList("E:\\file1.jar", 
   "E:\\file2.jar");
 FileOutputStream fos = new FileOutputStream("E:\\targetfile.zip");
 ZipOutputStream zipOut = new ZipOutputStream(fos);
 for (String srcFile: srcFiles) {
  File fileToZip = new File(srcFile);
  System.out.println(fileToZip + ", canRead:" + fileToZip.canRead());
  ZipEntry zipEntry = new ZipEntry(String.valueOf(srcFiles.indexOf(srcFile)) + "/" + fileToZip.getName());
  zipOut.putNextEntry(zipEntry);

  FileInputStream fis = new FileInputStream(fileToZip);
  IOUtils.copy(fis, zipOut);
  IOUtils.closeQuietly(fis);
 }
 IOUtils.closeQuietly(zipOut);
 IOUtils.closeQuietly(fos);
}

Scan log and paste to Google sheet to know the Flush pressure

Introduction

After a performance issue happen, we have less information about system status at that time.
There is flush log in Cassandra, so I write code to parse it and write to output file.
Thus I can paste the parse result to google sheet or excel to draw charts such as:

Code

public  class  ScanLog  {

  private  static  final  String  fileSufix  =  "10";
  private  static  final  String  LOG_FOLDER  =  "/cassandraLogPath";
  private  static  final  String  OUTPUT_FOLDER  =  "/outputpath";

  public  static  void  main(String[]  params)  throws  IOException,  ParseException  {
    File  folder  =  new  File(LOG_FOLDER);
    Map<String,List<CompactSize>>  cf2CompactSizes  =  toCompactSizes(folder);
    FileUtils.write(new  File(OUTPUT_FOLDER  +  "output-compact-mb-"  +  fileSufix  +  ".csv"),  toCompactSizesString(cf2CompactSizes),  "UTF-8");
    List<CompactSize>  aggCompactSizes  =  aggregate(toCompactSizeList(folder));
    FileUtils.write(new  File(OUTPUT_FOLDER  +  "output-compact-mb-agg-"  +  fileSufix  +  ".csv"),  toCompactSizesString(aggCompactSizes),  "UTF-8");
    List<FlushEvent>  events  =  toEvents(folder);
    FileUtils.write(new  File(OUTPUT_FOLDER  +  "output-queue-"  +  fileSufix  +  ".csv"),  toString(events),  "UTF-8");
    Map<String,  List<Enqueue>>  cfToE  =  cfToE(folder);
    Map<String,  LinkedHashMap<String,Long>>  cfToTimeToAvg  =  cfToTimeToAvg(cfToE);
    FileUtils.write(new  File(OUTPUT_FOLDER  +  "output-"  +  fileSufix  +  ".csv"),  toString(cfToE),  "UTF-8");
    FileUtils.write(new  File(OUTPUT_FOLDER  +  "output-"  +  fileSufix  +  "-avg.csv"),  toAvgString(cfToTimeToAvg),  "UTF-8");
  }

  private  static  List<CompactSize>  aggregate(List<CompactSize>  compactSizes)  {
    LinkedHashMap<String,List<CompactSize>>  minutes2CompactSizes  =  new  LinkedHashMap<>();
    compactSizes.forEach(compactSize  ->  {
      String  timeMinutes  =  compactSize.getTimeMinutes();
      if  (!minutes2CompactSizes.containsKey(timeMinutes))  {
        minutes2CompactSizes.put(timeMinutes,  new  ArrayList<>());
      }
      minutes2CompactSizes.get(timeMinutes).add(compactSize);
    });  
    LinkedHashMap<String,CompactSize>  minutes2AggregatedSize  =  new  LinkedHashMap<>();
    minutes2CompactSizes.entrySet().forEach(entry  ->  {
      String  minutes  =  entry.getKey();
      entry.getValue().forEach(size  ->  {
        if  (!minutes2AggregatedSize.containsKey(size.getTimeMinutes()))  {
          minutes2AggregatedSize.put(size.getTimeMinutes(),new  CompactSize(size.getTime(),size.getCf(),size.getDataSize()));
        }  else  {
          CompactSize  total  =  minutes2AggregatedSize.get(size.getTimeMinutes());
          CompactSize  newTotal  =  new  CompactSize(size.getTime(),size.getCf(),  size.getDataSize()  +  total.getDataSize());
          minutes2AggregatedSize.put(size.getTimeMinutes(),  newTotal);
        }
      });
    });
    List<CompactSize>  result  =  new  ArrayList<>();
    minutes2AggregatedSize.entrySet().forEach(entry  ->  {
      result.add(entry.getValue());
    });
    Collections.sort(result,  Comparator.comparingLong(CompactSize::getTimeMillis));
    return  result;
  }
  
  private  static  String  toCompactSizesString(List<CompactSize>  cf2compactSizes)  {
    StringBuilder  sb  =  new  StringBuilder();
    cf2compactSizes.forEach(size  ->  {
      sb.append(size.getCf()  +  ","  +  size.getTime()  +  ","  +  size.getDataSizeInMB()).append("\n");
    });
    return  sb.toString();
  }
  
  private  static  String  toCompactSizesString(Map<String,List<CompactSize>>  cf2compactSizes)  {
    StringBuilder  sb  =  new  StringBuilder();
    cf2compactSizes.keySet().forEach(cf  ->  {
      cf2compactSizes.get(cf).forEach(size  ->  {
        sb.append(size.getCf()  +  ","  +  size.getTime()  +  ","  +  size.getDataSizeInMB()).append("\n");
      });
    });
    return  sb.toString();
  }

  private  static  List<CompactSize>  toCompactSizeList(File  folder)  {
    List<CompactSize>  result  =  new  ArrayList<>();
    Arrays.stream(folder.listFiles())
        .filter(file  ->  file.getName().contains("system.log"))
        .forEach(file  ->  {
          try  {
            for  (String  line:  FileUtils.readLines(file))  {
              if  (  CompactSize.isMyPattern(line)  )  {
                CompactSize  compactSize  =  new  CompactSize(line);
                result.add(compactSize);
              }
            }
          }  catch  (Exception  ex)  {
            ex.printStackTrace();
          }
        });
    Collections.sort(result,  Comparator.comparingLong(CompactSize::getTimeMillis));
    return  result;
  }
  
  private  static  Map<String,List<CompactSize>>  toCompactSizes(File  folder)  {
    Map<String,List<CompactSize>>  cf2CompactSizes  =  new  HashMap<>();
    Arrays.stream(folder.listFiles())
        .filter(file  ->  file.getName().contains("system.log"))
        .forEach(file  ->  {
          try  {
            for  (String  line:  FileUtils.readLines(file))  {
              if  (  CompactSize.isMyPattern(line)  )  {
                CompactSize  compactSize  =  new  CompactSize(line);
                if  (!cf2CompactSizes.containsKey(compactSize.getCf()))  {
                  cf2CompactSizes.put(compactSize.getCf(),  new  ArrayList<>());
                }
                cf2CompactSizes.get(compactSize.getCf()).add(compactSize);
              }
            }
          }  catch  (Exception  ex)  {
            ex.printStackTrace();
          }
        });
    cf2CompactSizes.values().forEach(compactSizes  ->  
        Collections.sort(compactSizes,  Comparator.comparingLong(CompactSize::getTimeMillis)));
    return  cf2CompactSizes;
  }
  
  private  static  class  CompactSize  {
    private  final  String  time;
    private  final  String  cf;
    private  final  long  dataSize;
    
    CompactSize(String  time,  String  cf,  long  dataSize)  {
      this.time  =  time;
      this.cf  =  cf;
      this.dataSize  =  dataSize;
    }
    
    CompactSize(String  line)  {
      this.time  =  substringBetween(line,  "]  ",",");
      this.cf  =  substringBetween(line,  "/data/ng/db/data/wsg/",  "/");
      this.dataSize  =  NumberUtils.toInt(StringUtils.replace(substringBetween(line,  "].    ",  "  bytes"),",",""));
    }

    public  static  boolean  isMyPattern(String  line)  {
      return  StringUtils.contains(line,  "CompactionTask.java  (line  302)  Compacted")
          &&  StringUtils.contains(line,  "[/data/ng/db/data/wsg/");
    }
    
    public  long  getDataSizeInMB()  {
      return  dataSize  //  bytes  
          /  1024    //  KB
          /  1024;  //  MB
    }
    
    public  long  getDataSize()  {
      return  dataSize;
    }

    public  String  getCf()  {
      return  cf;
    }

    public  String  getTimeMinutes()  {
      long  millis  =  getTimeMillis();
      return  sdfToMinutes.format(new  Date(millis));
    }
    
    public  Long  getTimeMillis()  {
      return  toMillis(getTime());
    }
    
    public  String  getTime()  {
      return  time;
    }
  }
  
  private  static  String  toString(List<FlushEvent>  events)  {
    StringBuilder  sb  =  new  StringBuilder();
    AtomicInteger  waitForWrite  =  new  AtomicInteger();
    AtomicInteger  writing  =  new  AtomicInteger();
    events.forEach(e  ->  {
      switch  (  e.getType()  )  {
        case  Enqueue:
          waitForWrite.incrementAndGet();
          break;
        case  Write:
          writing.incrementAndGet();
          waitForWrite.decrementAndGet();
          break;
        case  Complete:
          writing.decrementAndGet();
          break;
        default:  
          throw  new  RuntimeException("Shouldn't  happen");
      }
      sb.append(e.getTime()  +  ","  +  waitForWrite  +  ","  +  writing).append("\n");
    });
    return  sb.toString();
  }
  
  private  static  List<FlushEvent>  toEvents(File  folder)  {
    List<FlushEvent>  events  =  new  ArrayList<>();
    Arrays.stream(folder.listFiles())
        .filter(file  ->  file.getName().contains("system.log"))
        .forEach(file  ->  {
          try  {
            for  (String  line:  FileUtils.readLines(file))  {
              Optional<FlushEventType>  type  =  FlushEventType.getByPattern(line);
              type.ifPresent(t  ->  {
                events.add(new  FlushEvent(line));
              });
            }
          }  catch  (Exception  ex)  {
            ex.printStackTrace();
          }
          });
    Collections.sort(events,  Comparator.comparingLong(FlushEvent::getTimeMillis));
    return  events;
  }

  private  static  String  toAvgString(Map<String,  LinkedHashMap<String,Long>>  cfToTimeToAvg)  {
    StringBuilder  sb  =  new  StringBuilder();
    cfToTimeToAvg.keySet().forEach(cf  ->  {
      cfToTimeToAvg.get(cf).forEach((time,avg)  ->  {
        sb.append(cf  +  ","  +  time  +  ","  +  avg).append("\n");
      });
    });
    return  sb.toString();
  }

  private  static  String  toString(Map<String,  List<Enqueue>>  cfToE)  {
    StringBuilder  sb  =  new  StringBuilder();
    cfToE.values().stream().forEach(eList  ->  {
      eList.stream().forEach(e  ->  {
        try  {
          sb.append(e.getCf()  +  ","  +  e.getTime()  +  ","  +  e.serializedMB).append("\n");
        }  catch  (Exception  ex)  {
          ex.printStackTrace();
        }
      });
    });
    return  sb.toString();
  }

  private  static  Map<String,  LinkedHashMap<String,Long>>  cfToTimeToAvg(Map<String,  List<Enqueue>>  cfToE)  {
    Map<String,  LinkedHashMap<String,Long>>  cfToTimeToAvg  =  new  HashMap<>();
    cfToE.keySet().forEach(cf  ->  {
      if  (!cfToTimeToAvg.containsKey(cf))  {
        cfToTimeToAvg.put(cf,  new  LinkedHashMap<>());
      }
      LinkedHashMap<String,List<Long>>  timeToMBList  =  new  LinkedHashMap<>();
      cfToE.get(cf).forEach(enqueue  ->  {
        if  (!timeToMBList.containsKey(enqueue.getTimeMinutes()))  {
          timeToMBList.put(enqueue.getTimeMinutes(),  new  ArrayList<>());
        }
        timeToMBList.get(enqueue.getTimeMinutes()).add(enqueue.serializedMB);
      });
      AtomicReference<String>  previousTime  =  new  AtomicReference<>();
      LinkedHashMap<String,Long>  timeToAvg  =  cfToTimeToAvg.get(cf);
      timeToMBList.keySet().forEach(time  ->  {
        try  {
          if  (previousTime.get()  !=  null)  {
            AtomicLong  totalMB  =  new  AtomicLong();
            timeToMBList.get(time).forEach(mb  ->  {
              totalMB.addAndGet(mb);
            });
            long  currentTimeMillis  =  sdfToMinutes.parse(time).getTime();
            long  previousTimeMillis  =  sdfToMinutes.parse(previousTime.get()).getTime();
            long  gapMinutes  =  TimeUnit.MILLISECONDS.toMinutes(currentTimeMillis  -  previousTimeMillis);
            timeToAvg.put(time,  (totalMB.longValue()  /  gapMinutes));
          }
          previousTime.set(time);
        }  catch  (Exception  ex)  {
          ex.printStackTrace();
        }
      });
    });
    return  cfToTimeToAvg;
  }

  private  static  Map<String,  List<Enqueue>>  cfToE(File  folder)  {
    Map<String,List<Enqueue>>  cfToE  =  new  HashMap<>();
    Arrays.stream(folder.listFiles())
        .filter(f  ->  f.getName().contains("system.log"))
        .forEach(file  ->  {
          try  {
            List<String>  lines  =  IOUtils.readLines(new  FileInputStream(file));
            for  (String  line:  lines)  {
              if  (StringUtils.contains(line,  "Enqueuing  flush  of  Memtable-"))  {
                Enqueue  e  =  new  Enqueue(line);
                if  (e.serializedMB  <=  1)  {
                  continue;
                }
                if  (!cfToE.containsKey(e.getCf()))  {
                  cfToE.put(e.getCf(),  new  ArrayList<>());
                }
                cfToE.get(e.getCf()).add(e);
              }
            }
          }  catch  (Exception  ex)  {
            ex.printStackTrace();
          }
        });
    cfToE.keySet().forEach(k  ->  {
      cfToE.get(k).sort((e1,e2)  ->  {
        try  {
          return  (int)(e1.getTimeMillis()-e2.getTimeMillis());
        }  catch  (Exception  ex)  {
          ex.printStackTrace();
          return  0;
        }
      });
    });
    return  cfToE;
  }

  private  enum  FlushEventType  {
    Enqueue("Enqueuing  flush  of  Memtable-"),
    Write("Writing  Memtable-"),
    Complete("Completed  flushing  ");
    private  final  String  pattern;
    FlushEventType(String  pattern)  {
      this.pattern  =  pattern;
    }

    public  String  getPattern()  {
      return  pattern;
    }

    public  static  Optional<FlushEventType>  getByPattern(String  line)  {
      return  Arrays.stream(values())
          .filter(type  ->  StringUtils.contains(line,  type.getPattern()))
          .findFirst();
    }
  }
  
  private  static  class  FlushEvent  {
    private  final  FlushEventType  type;
    private  final  String  time;
    FlushEvent(String  line)  {
      this.time  =  substringBetween(line,  "]  ",",");
      this.type  =  FlushEventType.getByPattern(line).get();  //  must  success
    }

    public  FlushEventType  getType()  {
      return  type;
    }

    public  Long  getTimeMillis()  {
      return  toMillis(getTime());
    }
    
    public  String  getTime()  {
      return  time;
    }
  }
  
  private  static  class  Enqueue  {
    private  final  String  time;
    private  final  String  cf;
    private  final  int  ops;
    private  final  long  serializedMB;

    Enqueue(String  line)  throws  ParseException  {
      this.time  =  substringBetween(line,  "]  ",",");
      this.cf  =  substringBetween(line,  "Enqueuing  flush  of  Memtable-",  "@");
      this.ops  =  NumberUtils.toInt(substringBetween(line,  "serialized/live  bytes,  ",  "  ops)"));
      this.serializedMB  =  NumberUtils.toInt(substringBetween(substringAfter(line,  "Enqueuing  flush  of  Memtable-"),  "(",  "/"))  /  1024  /  1024;
    }

    @Override
    public  String  toString()  {
      return  new  ToStringBuilder(this)
          .append("time",time)
          .append("cf",cf)
          .append("ops",ops)
          .append("serializedMB",  serializedMB)
          .toString();
    }

    public  String  getTimeMinutes()  {
      long  millis  =  getTimeMillis();
      return  sdfToMinutes.format(new  Date(millis));
    }

    public  Long  getTimeMillis()  {
      return  toMillis(getTime());
    }

    public  String  getTime()  {
      return  time;
    }

    public  String  getCf()  {
      return  cf;
    }

  }

  private  static  final  SimpleDateFormat  sdfToMinutes  =  new  SimpleDateFormat("yyyy-MM-dd  HH:mm");
  private  static  final  SimpleDateFormat  sdf  =  new  SimpleDateFormat("yyyy-MM-dd  HH:mm:SS");

  private  static  long  toMillis(String  time)  {
    try  {
      Date  date  =  sdf.parse(time);
      return  date.getTime();
    }  catch  (Exception  ex)  {
      throw  new  RuntimeException(ex);
    }
  }

}

lombok + Orika + Builder Pattern

Reference


Introduction

  1. Builder Pattern is suggested in Effective Java and used to generate immutable object.
  2. Builder Pattern introduces many codes, lombok project makes it easier
  3. We often use Orika to map data when transfer object from UI entity to domain entity or data persistence object to domain entity.
  4. It's hard to map Java Bean to Builder Pattern object because Builder Pattern doesn't follows Java Bean convention. It causes we can't leverage lombok and Orika at the same time

How to fix

  1. Copy and paste the following code: BuilderPropertyResolver
  2. /**
     * This class is used to map from an object to a builder who follows Builder Pattern.
     * 
     * <pre>
     * MapperFactory factory = new DefaultMapperFactory.Builder()
     *  .propertyResolverStrategy(new BuilderPropertyResolver())
     *  .build();
     *  factory.registerObjectFactory((Object o, MappingContext mappingContext) 
     *  -&gt; new MyBuilder(), TypeFactory.valueOf(MyBuilder.class));
     *  </pre>
     */
    public class BuilderPropertyResolver extends IntrospectorPropertyResolver {
    
     private static final Logger logger = LoggerFactory.getLogger(BuilderPropertyResolver.class);
     
     @Override
     protected void collectProperties(Class<?> type, Type<?> referenceType, Map<String, Property> properties) {
      super.collectProperties(type, referenceType, properties);
      if (StringUtils.endsWith(type.getName(), "Builder")) {
       Set<String> fieldNames = Arrays.stream(type.getDeclaredFields()).map(Field::getName).collect(Collectors.toSet());
       Arrays.stream(type.getDeclaredMethods())
         .filter(method -> fieldNames.contains(method.getName()))
         .filter(method -> isWriteMethodInBuilder(type, method))
         .forEach(method -> {
          Property.Builder builder = new Property.Builder();
          builder.expression(method.getName());
          builder.name(method.getName());
          builder.setter(method.getName() + "(%s)");
          Class<?> fieldType = method.getParameterTypes()[0];
          builder.type(this.resolvePropertyType(null, fieldType, type, referenceType));
          Property property = builder.build(this);
          properties.put(method.getName(), property);
         });
      }
     }
    
     private boolean isWriteMethodInBuilder(Class<?> type, Method method) {
      try {
       Class<?> returnType = method.getReturnType();
       Class<?> fieldType = type.getDeclaredField(method.getName()).getType();
       boolean onlyOneParameter = method.getParameterTypes().length == 1;
       boolean methodReturnBuilderType = returnType.equals(type);
       Class<?> methodParamType = method.getParameterTypes()[0];
       boolean isFieldSetter = fieldType.equals(methodParamType);
       return methodReturnBuilderType && onlyOneParameter && isFieldSetter;
      } catch (NoSuchFieldException e) {
       e.printStackTrace();
       return false;
      }
     }
    
    }
    
  3. Leverage this object
    public class Mapper {
     
     public static void main(String[] params) {
      MapperFactory factory = new DefaultMapperFactory.Builder()
        .propertyResolverStrategy(new BuilderPropertyResolver())
        .build();
      factory.registerObjectFactory((Object o, MappingContext mappingContext) 
        -> Person.builder(), TypeFactory.valueOf(Person.PersonBuilder.class));
      
      
      PersonBO bo = new PersonBO();
      bo.setName("TEST");
      factory.classMap(PersonBO.class, Person.class);
      System.out.println(factory.getMapperFacade().map(bo, Person.PersonBuilder.class).build());
      
      Person.PersonBuilder builder = Person.builder().list(Arrays.asList("A"));
      factory.getMapperFacade().map(bo, builder);
      System.out.println(builder);
     }
    
     @Data
     public static class PersonBO {
      private String name;
      private List<String> list;
     }
     
     @ToString
     @Builder
     public static class Person {
      private String name;
      private List<String> list;
     }
     
    }
    

Handle InterruptedException

Reference


How to handle InterruptedException

  • Throw it directly
  • If you can't throw it, call Thread.currentThread().interrupt()
  • Check Thread.currentThread().isInterrupted() every while loop if you catch an InterruptedException within the loop block

Example:

This example shows how to handle InterruptedException, you can change code to check what will happen in other cases.
public class TestMain {

 public static void main(String[] params) {
  ThreadPoolExecutor e = (ThreadPoolExecutor) Executors.newCachedThreadPool();
  MyRunnable r = new MyRunnable();
  e.execute(r);
  System.out.println("shutdown");
  e.shutdownNow();
  System.out.println("shutdown done");
 }

 private static class MyRunnable implements Runnable {

  @Override
  public void run() {
   while (!Thread.currentThread().isInterrupted()) {
    try {
     System.out.println("sleep");
     TimeUnit.HOURS.sleep(1);
    } catch (InterruptedException e) {
     e.printStackTrace();
     Thread.currentThread().interrupt();
    }
   }
  }
 }

}

Java TLS Practice

Introduction

記錄 Java 使用 SSL 的方法 (不要求 client auth)

Reference

Prepare jks files

  1. Install openssl in Ubuntu
    sudo apt-get install openssl
  2. Execute following commands one by one
    openssl req -x509 -newkey rsa:1024 -keyout private/cakey.pem -out private/cacert.pem -days 3650
    openssl x509 -in private/cacert.pem -addtrust clientAuth -setalias "Isaac Test Class 1 CA" -out public/catrust.pem
    keytool -importcert -trustcacerts -noprompt -file private/cacert.pem -alias ca -keystore public/catrust.jks -storepass 123456
    keytool -genkeypair -keyalg RSA -keysize 1024 -validity 730 -keystore public/server.jks
    keytool -certreq -file server-req.pem -keystore public/server.jks
    openssl x509 -req -in server-req.pem -out public/server-cert.pem -CA private/cacert.pem -CAkey private/cakey.pem -extensions v3_usr -days 730 -CAserial private/cacert.srl -CAcreateserial
    cat private/cacert.pem >> public/server-cert.pem
    keytool -importcert -v -file public/server-cert.pem -keystore public/server.jks
    rm server-req.pem
  3. There are public/server.jks and public/catrust.jks

Run server code

public class ServerDontNeedClientAuth {

    private static String SERVER_KEY_STORE = Paths.get("src/main/resources/certs/server.jks").toAbsolutePath().toString();
    private static String SERVER_KEY_STORE_PASSWORD = "123456";

    public static void main(String[] params) throws Exception {
//        System.setProperty("javax.net.debug", "ssl,handshake");
        System.setProperty("javax.net.ssl.trustStore", SERVER_KEY_STORE);
        SSLContext context = SSLContext.getInstance("TLS");
        KeyStore ks = KeyStore.getInstance("jceks");
        ks.load(new FileInputStream(SERVER_KEY_STORE), null);
        KeyManagerFactory kf = KeyManagerFactory.getInstance("SunX509");
        kf.init(ks, SERVER_KEY_STORE_PASSWORD.toCharArray());
        context.init(kf.getKeyManagers(), null, null);

        ServerSocketFactory factory = context.getServerSocketFactory();
        ServerSocket serverSocket = factory.createServerSocket(8443);
        SSLServerSocket sslServerSocket =  (SSLServerSocket) serverSocket;
        sslServerSocket.setNeedClientAuth(false);
        while(true){
            try{
                System.out.println("listen port 8443..");
                Socket socket = sslServerSocket.accept();
                System.out.println("accept:" + socket);
                InputStream is = socket.getInputStream();
                BufferedReader buffer = new BufferedReader(new InputStreamReader(is));
                String readLine = buffer.readLine();
                System.out.println("server receive:" + readLine);
                socket.getOutputStream().write("1234567890".getBytes());
                socket.close();
            }catch(Exception e){
                e.printStackTrace();
            }
        }
    }

}

Run client code

public class ClientWithoutAuth {

    private static String CLIENT_KEY_STORE = Paths.get("src/main/resources/certs/catrust.jks").toAbsolutePath().toString();

    public static void main(String[] params) throws Exception {
        System.setProperty("javax.net.ssl.trustStore", CLIENT_KEY_STORE);
        SocketFactory sf = SSLSocketFactory.getDefault();
        Socket s = sf.createSocket("localhost", 8443);
        PrintWriter writer = new PrintWriter(s.getOutputStream());
        BufferedReader reader = new BufferedReader(new InputStreamReader(s.getInputStream()));
        writer.println("hello\n");
        writer.flush();
        System.out.println("client receive:" + reader.readLine());
        s.close();
    }

}

Server output

listen port 8443..
accept:340f438e[SSL_NULL_WITH_NULL_NULL: Socket[addr=/127.0.0.1,port=50429,localport=8443]]
server receive:hello
listen port 8443..

Client output

1234567890

guava notes

Reference

Optional

Optional 可以用來處理 null value 與避免 NullPointerException
public class OptionalTest {


    @Test
    public void test() {
        try {
            Optional.of(null);
            Assert.fail("Optional.of shouldn't accept null");
        } catch (NullPointerException ex) {
        }
        Optional opt = Optional.of("Test");
        Assert.assertTrue(opt.isPresent());
        Assert.assertEquals("Test", opt.get());
        Assert.assertEquals("Test", opt.or("QQ"));
        Assert.assertEquals("Test", opt.orNull());
        Assert.assertEquals(ImmutableSet.of("Test"), opt.asSet());

        Optional nullableOptional = Optional.fromNullable(null);
        Assert.assertFalse(nullableOptional.isPresent());
        try {
            Assert.assertNull(nullableOptional.get());
            Assert.fail();
        } catch (IllegalStateException ex) {
        }
        Assert.assertEquals("QQ", nullableOptional.or("QQ"));
        Assert.assertNull(nullableOptional.orNull());
        Assert.assertEquals(ImmutableSet.of(), nullableOptional.asSet());
        Assert.assertEquals(Optional.absent(), nullableOptional);
    }

}

ComparisonChain

ComparisonChain 可以用比較漂亮的程式實作 compare
public class ComparisonChainTest {

    @Test
    public void test() {
        NumbersObject n1 = new NumbersObject(1,2,3);
        NumbersObject n2 = new NumbersObject(1,2,4);
        NumbersObject n3 = new NumbersObject(1,3,4);
        NumbersObject n4 = new NumbersObject(2,2,4);

        List<NumbersObject> list = Arrays.asList(n4, n3, n2, n1);
        Collections.sort(list, (o1, o2) -> ComparisonChain.start()
                .compare(o1.a,o2.a)
                .compare(o1.b,o2.b)
                .compare(o1.c,o2.c)
                .result());
        Assert.assertTrue(n1 == list.get(0));
        Assert.assertTrue(n2 == list.get(1));
        Assert.assertTrue(n3 == list.get(2));
        Assert.assertTrue(n4 == list.get(3));
    }

    final static class NumbersObject {
        private final int a ;
        private final int b;
        private final int c;
        NumbersObject(int a, int b, int c) {
            this.a = a;
            this.b = b;
            this.c = c;
        }
    }

}

Ordering

Ordering 其實就是 Comparator, 可以傳給 Collections. 例如: Collections.sort(list, Ordering)

usingToString

用 toString 的結果排序
@Test
public void usingToString() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(1, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4);
    Collections.sort(numbers, Ordering.usingToString());
    Assert.assertEquals(n3, numbers.get(0));
    Assert.assertEquals(n4, numbers.get(1));
    Assert.assertEquals(n1, numbers.get(2));
    Assert.assertEquals(n2, numbers.get(3));
}

private final class NumbersObject {

    public final Integer a;
    public final Integer b;
    public final Integer c;

    public NumbersObject(Integer a, Integer b, Integer c) {
        this.a = a;
        this.b = b;
        this.c = c;
    }

    @Override
    public String toString() {
        return MoreObjects.toStringHelper(this)
                .add("a",a)
                .add("b",b)
                .add("c",c)
                .toString();
    }
}

nullsFirst & nullsLast

排序的時候, null 放在最前面位置(或最後).
物件本身是 Comparable, 可以直接拿 nullsFirst 回傳的 Ordering 來排序, 若物件本身不是 Comparable 就要提供排序的方式
@Test
public void nullsFirst_NotComparable() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(1, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    NumbersObject nullNumber = new NumbersObject(null,null,null);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4, nullNumber);
    Ordering ordering = Ordering.natural().nullsFirst().onResultOf(new Function<NumbersObject, Integer>() {
        @Override
        public Integer apply(NumbersObject numbersObject) {
            return numbersObject.a;
        }
    });
    Collections.sort(numbers, ordering);
    Assert.assertEquals(nullNumber, numbers.get(0));
    Assert.assertEquals(n4, numbers.get(1));
    Assert.assertEquals(n3, numbers.get(2));
    Assert.assertEquals(n1, numbers.get(3));
    Assert.assertEquals(n2, numbers.get(4));
}

@Test
public void nullsFirst_Comparable() {
    List<Integer> list = Arrays.asList(5, 2, 6, 7, 3, null);
    Collections.sort(list, Ordering.natural().nullsFirst());
    Assert.assertEquals((Integer) null, list.get(0));
    Assert.assertEquals((Integer) 2, list.get(1));
    Assert.assertEquals((Integer) 3, list.get(2));
    Assert.assertEquals((Integer) 5, list.get(3));
    Assert.assertEquals((Integer) 6, list.get(4));
    Assert.assertEquals((Integer) 7, list.get(5));
}

@Test
public void nullsLast_Comparable() {
    List<Integer> list = Arrays.asList(5, 2, 6, 7, 3, null);
    Collections.sort(list, Ordering.natural().nullsLast());
    Assert.assertEquals((Integer) 2, list.get(0));
    Assert.assertEquals((Integer) 3, list.get(1));
    Assert.assertEquals((Integer) 5, list.get(2));
    Assert.assertEquals((Integer) 6, list.get(3));
    Assert.assertEquals((Integer) 7, list.get(4));
    Assert.assertEquals((Integer) null, list.get(5));
}

@Test
public void nullsLast_NotComparable() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(1, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    NumbersObject nullNumber = new NumbersObject(null,null,null);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4, nullNumber);
    Ordering ordering = Ordering.natural().nullsLast().onResultOf(new Function<NumbersObject, Integer>() {
        @Override
        public Integer apply(NumbersObject numbersObject) {
            return numbersObject.a;
        }
    });
    Collections.sort(numbers, ordering);
    Assert.assertEquals(n4, numbers.get(0));
    Assert.assertEquals(n3, numbers.get(1));
    Assert.assertEquals(n1, numbers.get(2));
    Assert.assertEquals(n2, numbers.get(3));
    Assert.assertEquals(nullNumber, numbers.get(4));
}

private final class NumbersObject {

    public final Integer a;
    public final Integer b;
    public final Integer c;

    public NumbersObject(Integer a, Integer b, Integer c) {
        this.a = a;
        this.b = b;
        this.c = c;
    }

    @Override
    public String toString() {
        return MoreObjects.toStringHelper(this)
                .add("a",a)
                .add("b",b)
                .add("c",c)
                .toString();
    }
}

compound

可以組合多個 Comparator, 在上一個 Comparator 無法決定時就交給下一個 Comparator 判斷.
在排列的屬性有優先順序的時候可以讓程式變簡單.
例如: 先排 a, 再 b, 再 c. null 的話就排前面
@Test
public void compound() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(1, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    NumbersObject nullNumber = new NumbersObject(null,null,null);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4, nullNumber);
    Comparator<NumbersObject> byNullFirstA = Ordering.natural().nullsFirst().onResultOf((t) -> t.a);
    Comparator<NumbersObject> byNullFirstB = Ordering.natural().nullsFirst().onResultOf((t) -> t.b);
    Comparator<NumbersObject> byNullFirstC = Ordering.natural().nullsFirst().onResultOf((t) -> t.c);
    Ordering orderByABC = Ordering.compound(Arrays.asList(byNullFirstA, byNullFirstB, byNullFirstC));
    Collections.sort(numbers, orderByABC);
    Assert.assertEquals(nullNumber, numbers.get(0));
    Assert.assertEquals(n4, numbers.get(1));
    Assert.assertEquals(n3, numbers.get(2));
    Assert.assertEquals(n1, numbers.get(3));
    Assert.assertEquals(n2, numbers.get(4));
}

private final class NumbersObject {

    public final Integer a;
    public final Integer b;
    public final Integer c;

    public NumbersObject(Integer a, Integer b, Integer c) {
        this.a = a;
        this.b = b;
        this.c = c;
    }

    @Override
    public String toString() {
        return MoreObjects.toStringHelper(this)
                .add("a",a)
                .add("b",b)
                .add("c",c)
                .toString();
    }
}

lexicographical

跟 Collections.sort 不同, loxicographical 是用來替很多個 iterable 作排序
@Test
public void lexicographically() {
    ImmutableList<String> empty = ImmutableList.of();
    ImmutableList<String> a = ImmutableList.of("a");
    ImmutableList<String> aa = ImmutableList.of("a", "a");
    ImmutableList<String> ab = ImmutableList.of("a", "b");
    ImmutableList<String> b = ImmutableList.of("b");
    List<ImmutableList<String>> all = Arrays.asList(b,a,ab,aa,empty);
    Ordering<Iterable<String>> c = Ordering.<String>natural().lexicographical();
    Collections.sort(all, c);
    Assert.assertEquals(empty, all.get(0));
    Assert.assertEquals(a, all.get(1));
    Assert.assertEquals(aa, all.get(2));
    Assert.assertEquals(ab, all.get(3));
    Assert.assertEquals(b, all.get(4));
}

@Test
public void lexicographicallyReverse() {
    ImmutableList<String> empty = ImmutableList.of();
    ImmutableList<String> a = ImmutableList.of("a");
    ImmutableList<String> aa = ImmutableList.of("a", "a");
    ImmutableList<String> ab = ImmutableList.of("a", "b");
    ImmutableList<String> b = ImmutableList.of("b");
    List<ImmutableList<String>> all = Arrays.asList(b,a,ab,aa,empty);
    Ordering<Iterable<String>> c = Ordering.<String>natural().lexicographical().reverse();
    Collections.sort(all, c);
    Assert.assertEquals(empty, all.get(4));
    Assert.assertEquals(a, all.get(3));
    Assert.assertEquals(aa, all.get(2));
    Assert.assertEquals(ab, all.get(1));
    Assert.assertEquals(b, all.get(0));
}

@Test
public void reverseLexicographically() {
    ImmutableList<String> empty = ImmutableList.of();
    ImmutableList<String> a = ImmutableList.of("a");
    ImmutableList<String> aa = ImmutableList.of("a", "a");
    ImmutableList<String> ab = ImmutableList.of("a", "b");
    ImmutableList<String> b = ImmutableList.of("b");
    List<ImmutableList<String>> all = Arrays.asList(b,a,ab,aa,empty);
    Ordering<Iterable<String>> c = Ordering.<String>natural().reverse().lexicographical();
    Collections.sort(all, c);
    System.out.println(all);
    Assert.assertEquals(empty, all.get(0));
    Assert.assertEquals(b, all.get(1));
    Assert.assertEquals(a, all.get(2));
    Assert.assertEquals(ab, all.get(3));
    Assert.assertEquals(aa, all.get(4));
}

onResultOf

就是自訂排序方式, 直行完後會再往前呼叫其他排序法
@Test
public void onResultOf() {
    Comparator<Integer> c = Ordering.<Integer>natural().onResultOf((x) -> Optional.ofNullable(x).orElse(4)%5);
    List<Integer> list = Arrays.asList(5, 11, 4, 1, 2, 5, 7, 9, null);
    Collections.sort(list, c);
    System.out.println(list);
    Assert.assertEquals(5, list.get(0).intValue());
    Assert.assertEquals(5, list.get(1).intValue());
    Assert.assertEquals(11, list.get(2).intValue());
    Assert.assertEquals(1, list.get(3).intValue());
    Assert.assertEquals(2, list.get(4).intValue());
    Assert.assertEquals(7, list.get(5).intValue());
    Assert.assertEquals(4, list.get(6).intValue());
    Assert.assertEquals(9, list.get(7).intValue());
    Assert.assertEquals(null, list.get(8));
}

@Test
public void onResultOf_reverse() {
    Comparator<Integer> c = Ordering.<Integer>natural().reverse().onResultOf((x) -> Optional.ofNullable(x).orElse(4)%5);
    List<Integer> list = Arrays.asList(5, 11, 4, 1, 2, 5, 7, 9, null);
    Collections.sort(list, c);
    System.out.println(list);
    Assert.assertEquals(5, list.get(8).intValue());
    Assert.assertEquals(5, list.get(7).intValue());
    Assert.assertEquals(11, list.get(5).intValue());
    Assert.assertEquals(1, list.get(6).intValue());
    Assert.assertEquals(2, list.get(3).intValue());
    Assert.assertEquals(7, list.get(4).intValue());
    Assert.assertEquals(4, list.get(0).intValue());
    Assert.assertEquals(9, list.get(1).intValue());
    Assert.assertEquals(null, list.get(2));
}

greatestOf 與 leastOf

拿排序過資料的最後幾筆與最前幾筆.
@Test
public void greatestOf() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(1, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    NumbersObject nullNumber = new NumbersObject(null,null,null);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4, nullNumber);
    List<NumbersObject> greatestOf = Ordering.natural().greatestOf(numbers,3);
    Assert.assertEquals(n4, greatestOf.get(0));
    Assert.assertEquals(n3, greatestOf.get(1));
    Assert.assertEquals(n1, greatestOf.get(2));
}

@Test
public void leastOf() {
    NumbersObject n1 = new NumbersObject(1, 2, 3);
    NumbersObject n2 = new NumbersObject(2, 2, 4);
    NumbersObject n3 = new NumbersObject(-1, 2, -3);
    NumbersObject n4 = new NumbersObject(-2, 3, 5);
    NumbersObject nullNumber = new NumbersObject(null,null,null);
    List<NumbersObject> numbers = Arrays.asList(n1, n2, n3, n4, nullNumber);
    List<NumbersObject> leastOf = Ordering.natural().leastOf(numbers,3);
    Assert.assertEquals(nullNumber, leastOf.get(0));
    Assert.assertEquals(n2, leastOf.get(1));
    Assert.assertEquals(n1, leastOf.get(2));
}

private final class NumbersObject implements Comparable<NumbersObject> {

    public final Integer a;
    public final Integer b;
    public final Integer c;

    public NumbersObject(Integer a, Integer b, Integer c) {
        this.a = a;
        this.b = b;
        this.c = c;
    }

    @Override
    public String toString() {
        return MoreObjects.toStringHelper(this)
                .add("a",a)
                .add("b",b)
                .add("c",c)
                .toString();
    }

    @Override
    public int compareTo(NumbersObject o) {
        if (a == null && o.a == null) {
            return 0;
        } else if (a == null) {
            return -1;
        } else if (o.a == null) {
            return 1;
        } else {
            return Ints.compare(o.a,a);
        }
    }
}

ImmutableXXX

沒有 ImmutableSet 的時候會用 Collections.unmodifiableSet, 差別是 ImmutableSet 有很多好用的 api 讓程式簡單很多.
另外 ImmutableXXX 在能夠避免線性成長的 copy time 時候會盡量避免. 例如 ImmutableList.copyOf(ImmutableSet)
@Test(expected = UnsupportedOperationException.class)
public void of() {
    ImmutableSet.of("a","b").add("a");
}

@Test(expected = UnsupportedOperationException.class)
public void copyOf() {
    ImmutableSet.copyOf(Arrays.asList("a","b","c")).add("");
}

@Test
public void listCopyOfSet() {
    ImmutableList immutableList = ImmutableList.copyOf(ImmutableSet.of("a","b"));
}

Multipleset

像是 set 一樣, 差別是一個同樣的資料可以存多筆. 也可以記錄某種資料總共有幾筆.
@RunWith(Parameterized.class)
public class MultisetTest {

    @Parameterized.Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][] {
                { HashMultiset.create() },
                {TreeMultiset.create() },
                {LinkedHashMultiset.create() },
                {ConcurrentHashMultiset.create()},
        });
    }

    private final Multiset<String> testee;

    public MultisetTest(Multiset<String> testee) {
        this.testee = testee;
    }

    @Test
    public void test() {
        testee.addAll(Arrays.asList("A","B","A"));
        Assert.assertEquals(2, testee.count("A"));
        Assert.assertEquals(1, testee.count("B"));
    }

}

public class ImmutableMultisetTest {

    @Test
    public void count() {
        ImmutableMultiset set = ImmutableMultiset.of("A","B","A");
        Assert.assertEquals(2,set.count("A"));
        Assert.assertEquals(1,set.count("B"));
    }

}

function 參數先排位置再對參數名稱

如果沒有指定要丟的參數名稱, python 就會照順序排. 如果順序對不起來就會 error.
>>> def f(a,b,c):
 print 'a:', a, 'b:', b, 'c:', c

 
>>> f(1,2,3)
a: 1 b: 2 c: 3
>>> f(1,c=3,b=2)
a: 1 b: 2 c: 3
>>> f(1,2,c=3)
a: 1 b: 2 c: 3
>>> f(1,2,b=2) # 第二個參數已經指定了, 所以就不能再指定 b

Traceback (most recent call last):
  File "", line 1, in 
    f(1,2,b=2)
TypeError: f() got multiple values for keyword argument 'b'

>>> f(c=3,2,a=1)
SyntaxError: non-keyword arg after keyword arg

>>> f(a=1,2,3)
SyntaxError: non-keyword arg after keyword arg

字串 format

帶入 dict 的值, 如果用 format api, dictionary 要帶 ** 去展開 key-value
>>> m = {'a':1.1234, "b":"qq"}
>>> '%(a)1.1f %(a)s %(b)s' %m
'1.1 1.1234 qq'


>>> '{a:1.1f} {a:s} {b:s}'.format(m) # illegal, must use **m

Traceback (most recent call last):
  File "", line 1, in 
    '{a:1.1f} {a:s} {b:s}'.format(m)
KeyError: 'a'

>>> '{a:1.1f} {a:s} {b:s}'.format(**m) # illegal, a is float type

Traceback (most recent call last):
  File "", line 1, in 
    '{a:1.1f} {a:s} {b:s}'.format(**m)
ValueError: Unknown format code 's' for object of type 'float'

>>> '{a:1.1f} {b:s}'.format(**m)
'1.1 qq'

tuple 就是不可變的 list

tuple 就是不可變的 list.
>>> list = ['c',[1,2,3]]
>>> tuple = ('c',(1,2,3))
>>> list[0]
'c'
>>> list[1]
[1, 2, 3]
>>> tuple[0]
'c'
>>> tuple[1]
(1, 2, 3)
>>> list[0] = 'q'
>>> list[0]
'q'
>>> list[1]
[1, 2, 3]
>>> tuple[0] = 'q'

Traceback (most recent call last):
  File "", line 1, in 
    tuple[0] = 'q'
TypeError: 'tuple' object does not support item assignment

function is an object

python 的 function 是一個物件, 所以可以放進 collection 中等以後被呼叫.
>>> def a():
 print 'a';

 
>>> def b():
 print 'b';

 
>>> def c():
 print 'c';

 
>>> list = [a,b,c];
>>> for f in list:
 f();

 
a
b
c

split large file

I use Notepad++ to check log and debug.
When log file is large, I need to split large file to smaller ones.
I can't find tool so write codes to split file in Java every time.
I use Python to write a tool to split file when studying python.
Welcome to use this tool if you also need to split large log file.

Shell 筆記


  1. cd /d %~dp0
    不管執行 script 時候路徑在哪, cd /d %~dp0 後 script 的路徑就到該 script 路徑下
    script, 放在 f:/test/test.bat
    cd /d %~dp0
    dir
    
    execute: f:/test/test.bat
    f:\>test\test.bat
    
    f:\>cd /d f:\test\
    
    f:\test>dir
     磁碟區 F 中的磁碟是 新增磁碟區
     磁碟區序號:  B0DF-F1C2
    
     f:\test 的目錄
    
    2015/07/14  上午 11:41    <DIR>          .
    2015/07/14  上午 11:41    <DIR>          ..
    2015/07/14  上午 11:42                16 test.bat
                   1 個檔案               16 位元組
                   2 個目錄  224,189,362,176 位元組可用
    

Git 筆記


  1. 從 branch merge 回 master, 導致 pom.xml conflict, 想 reset pom.xml 因為這不是我要 merge 的內容. (stackoverflow)
     git reset pom.xml
     git checkout pom.xml
    
  2. 想清掉 untrack file
    git clean -f
    
  3. 想清掉 untrack folder
    git clean -f -d
    
  4. 有次一個 branch 太久沒 pull 了, 後來不知道誰改了甚麼, 要再 pull 都 conflict 一堆.
    反正我也沒有要保留 local 的東西, 同事就教我強制把 local 的檔案 reset 到某個版本
    git reset comm_id --hard
    
  5. 在 pull 之前就 commit, git status 出現 "Your branch is ahead of 'origin/master' by 25 commits" 的訊息, 用 reset 還原
    git reset --hard origin/master
    
  6. 要把一個 branch 傳送到另一個 repository
    # new remote
    git remote add remotename git@git.abc.com:test/test.git
    
    # push branch to remote
    git push -u remotename branchname
    

Basic Perl 筆記


  1. 單引號的 \n 不會換行, 雙引號內的 \n 會換行
    print("line1\nline2\n\n\n\n");
    print('line3:abc\nabc');
    
    d:\>test.pl
    input string:abc
    input number:4
    abcabcabcabc
    d:\>test.pl
    equal:1
    d:\>test.pl
    line1
    line2
    
    
    
    line3:abc\nabc
    
  2. 用 . 連字串
    $a = 'abc\nabc';
    $b = "def\ndef";
    $c = $a . $b . "QQQ";
    print("$c");
    
    d:\>test.pl
    abc\nabcdef
    defQQQ
    
  3. 數字 0 是 false, 其他都是 true.
    空字串是 false, 其他都是 true.
    undef 是 false.
    print($abc == 0); # print 1, $abc is undef and is false, false is 0
    $abc = '';
    print($abc == 0); # print 1, $abc is empty string and is false, false is 0
    $abc = "0";
    print($abc == 0); # print 1, $abc is string 0, Perl transfer to number 0
    
    d:\>test.pl
    111
    
  4. 使用 lt,le,eq,ge,gt 作字串的比較, perl 會用 ASCII 或 Unicode 作為順序參考排大小.
    $t1 = "a";
    $t2 = "a";
    if ($t1 eq $t2) {
        print("same");
    } else {
        print("different");
    }
    
    d:\>test2.pl
    same
    
    $t1 = "a";
    $t2 = "b";
    if ($t1 eq $t2) {
        print("same");
    } else {
        print("different");
    }
    
    d:\>test2.pl
    different
    
  5. 取得使用者輸入: <STDIN>
    print("What's your name?\n");
    $name = <STDIN>;
    print("$name, how are you?");
    
  6. d:\>test.pl
    What's your name?
    isaac
    isaac
    , how are you?
    



  7. 這時候就發現 <STDIN> 取得的資料會包含換行, 換行字元是不需要的, 就用 chomp 去掉
    print("What's your name?\n");
    $name = <STDIN>;
    chomp($name);
    print("$name, how are you?");
    

  8. d:\>test.pl
    What's your name?
    isaac
    isaac, how are you?
    



  9. 如果輸入的時候按 Ctrl + C, 沒有輸入, <STDIN> 會回傳 undef. 這時候可以用 defined 來判斷是否為 undef
    $name = <STDIN>;
    if (defined($name)) {
     chomp($name);
     print("input:$name");
    } else {
     print("no input\n");
    }
    

  10. d:\>test.pl
    no input
    Terminating on signal SIGINT(2)
    

  11. 使用陣列
    while ($i < 10) {
     $i += 1;
     $names[$i] = "p$i";
     print("$names[$i]\n");
    }
    $names[100] = "qq";
    print("$names[100]");
    
    d:\>test.pl
    p1
    p2
    p3
    p4
    p5
    p6
    p7
    p8
    p9
    p10
    qq
    
  12. 取陣列最後一個值的 index: $#names
    $names[100] = "aaa";
    print("$#names\n"); 
    print($names[$#names]); 
    
    d:\>test.pl
    100
    aaa
    
  13. 列出陣列全部值 (這裡是要注意 $#names 的值是最後一個 index 而不是長度. )
    想得到長度要再加 1, 因為還有第 0 個.
    $i = 0;
    while ($i < 10) {
     $i += 1;
     $names[$i] = $i;
    }
    $i = 0;
    while ($i <= $#names) {
     $i += 1;
     print("$names[$i]");
    }
    
    d:\>test.pl
    12345678910
    
  14. index 可以指定負數, -1 就是最後一個值
    $names[0]="0";
    $names[1]="1";
    $names[2]="2";
    $names[3]="3";
    print($names[0]); #0
    print($names[-1]); #3
    print($names[-2]); #2
    print($names[-3]); #1
    print($names[-4]); #0
    
  15. 用 @串列變數 = (用逗號分隔的串列值) 宣告串列
    會置換變數, \n 會換行, 就跟雙引號宣告的變數一樣
    @a = (1,2,3); 
    print("\@a:@a\n");
    
    @b = (1..3); # 使用 .. 會 +1
    print("\@b:@b\n");
    
    @c = (1.4...5.6); # 使用 .. 會無條件捨去小數
    print("\@c:@c\n");
    
    @d = (2,6...10,43); 
    print("\@d:@d\n");
    
    $e = 10;
    $f = 20;
    @g = ($e...$f); 
    print("\@g:@g\n");
    
    @h = ("a", "b\n", "c");  #有換行效果
    print("\@h:@h\n");
    
    d:\>test.pl
    @a:1 2 3
    @b:1 2 3
    @c:1 2 3 4 5
    @d:2 6 7 8 9 10 43
    @g:10 11 12 13 14 15 16 17 18 19 20
    @h:a b
     c
    
  16. 用 @串列變數=qw(用空白分隔的串列值) 宣告串列,
    不會置換變數, \n 不會換行, 跟單引號宣告的變數一樣
    @a = qw(1 2 3); #可以用 qw()
    print("\@a:@a\n"); 
    
    @b = qw<1 data-blogger-escaped-..3="">; #也可以 qw<>, 但 1..3 會直接印出來
    print("\@b:@b\n");
    
    @c = qw/1.4...5.6/; #也可以 qw//
    print("\@c:@c\n");
    
    @d = qw!2 6...10 43!; 
    print("\@d:@d\n");
    
    $e = 10;
    $f = 20;
    @g = qw($e...$f); #沒有換變數的效果
    print("\@g:@g\n");
    
    @h = qw("a" "b\n" "c");  #無換行效果, 雙引號也會被印出來
    print("\@h:@h\n");
    
    d:\>test.pl
    @a:1 2 3
    @b:1..3
    @c:1.4...5.6
    @d:2 6...10 43
    @g:$e...$f
    @h:"a" "b\n" "c"
    
  17. 一次 assign 值給多個變數
    ($a,$b,$c) = (1,2,3);
    print("a:$a\n");
    print("b:$b\n");
    print("c:$c\n");
    
    d:\>test.pl
    a:1
    b:2
    c:3
    
  18. 換值
    $a[0] = 0;
    $a[1] = 1;
    ($a[0],$a[1]) = ($a[1],$a[0]);
    print("a[0]:$a[0]\n");
    print("a[1]:$a[1]\n");
    
    d:\>test.pl
    a[0]:1
    a[1]:0
    
  19. 如果移除括號也不改變原本意思, 就可以移除括號
    @a = 1...3;
    print(@a);
    
    d:\>test.pl
    123
    
  20. 如果一個串列值是另一個串列, 被包含在串列裡的會被展開
    @a = 1..3;
    $b = 4;
    @c = ();
    #d is undefined
    @e = (@a,$b,@c,@d);
    print(@e);
    
    d:\>test.pl
    1234
    
  21. 用 pop 可以從串列取值出來, 沒有值的話會取出 undef
    @a = 1..3;
    while (defined($val = pop(@a))) {
     print("$val\n");
    }
    
    d:\>test.pl
    3
    2
    1
    
  22. 用 push 可以把值放進串列
    @a = 1..3;
    push(@a,4);
    while (defined($val = pop(@a))) {
     print("$val\n");
    }
    
    d:\>test.pl
    4
    3
    2
    1
    
  23. 串列可以複製全部的值 (不是 reference, 所以複製後對串列的修改不會互相影響)
    @a = 1...3;
    @b = @a;
    print("a:\n");
    while (defined($val = pop(@a))) {
     print("$val\n");
    }
    print("b:\n");
    while (defined($val = pop(@b))) {
     print("$val\n");
    }
    
    d:\>test.pl
    a:
    3
    2
    1
    b:
    3
    2
    1
    
  24. 一次 assign 值給多個變數
    ($a,$b,$c) = (1,2,3);
    print("a:$a\n");
    print("b:$b\n");
    print("c:$c\n");
    
    d:\>test.pl
    a:1
    b:2
    c:3
    
  25. qw 透過空白來區分值, 也可以 assign 值給多個變數
    ($google,$yahoo,$linkedin) = qw {
     http://www.google.com
     http://www.yahoo.com
     http://www.linkedin.com
    };
    print("google:$google\n");
    print("yahoo:$yahoo\n");
    print("linkedin:$linkedin\n");
    
    ($google,$yahoo,$linkedin) = qw !
     http://www.google.com
     http://www.yahoo.com
     http://www.linkedin.com
    !;
    print("google:$google\n");
    print("yahoo:$yahoo\n");
    print("linkedin:$linkedin\n");
    
    d:\>test.pl
    google:http://www.google.com
    yahoo:http://www.yahoo.com
    linkedin:http://www.linkedin.com
    google:http://www.google.com
    yahoo:http://www.yahoo.com
    linkedin:http://www.linkedin.com
    
  26. shift 從 index 0 取值, unshift 從 index 0 放值
    @a = ();
    unshift(@a,"1");
    unshift(@a,"2");
    unshift(@a,"3");
    print("@a\n"); #321
    print(shift(@a)); #3
    print(shift(@a)); #2
    print(shift(@a)); #1
    
  27. pop 從 index 最後取值, push 從 index 最後放值
    @a = ();
    unshift(@a,"1");
    unshift(@a,"2");
    unshift(@a,"3");
    print("@a\n"); #321
    push(@a,"4"); #3214
    push(@a,"5"); #32145
    push(@a,"6"); #321456
    print(shift(@a)); #3
    print(shift(@a)); #2
    print(shift(@a)); #1
    print(pop(@a)); #6
    print(pop(@a)); #5
    print(pop(@a)); #4
    
    d:\>test.pl
    3 2 1
    321654
    
  28. pop,push,shift,unshift 可以一次處理整個串列
    @a = ();
    unshift(@a, qw/ 1 2 3 /); #123
    unshift(@a, qw/ 4 5 6 /); #456123
    push(@a, qw/ 7 8 9 /); #456123789
    push(@a, qw/ 10 11 12 /); #456123789101112
    print(shift(@a)); #4
    print(shift(@a)); #5
    print(shift(@a)); #6
    print(shift(@a)); #1
    print(shift(@a)); #2
    print(shift(@a)); #3
    print(pop(@a)); #12
    print(pop(@a)); #11
    print(pop(@a)); #10
    print(pop(@a)); #9
    print(pop(@a)); #8
    print(pop(@a)); #7
    defined(pop(@a)) ? print("value") : print(" no val"); # no val
    
    d:\>test.pl
    456123121110987 no val
    
  29. 切串列: splice
    @a = 1..9;
    splice(@a,1); # 從 index 1 之後全切掉
    print("@a\n"); #1
    @a = 1..9;
    @removed = splice(@a,1,3); # 從 index 1 切掉三個
    print("@a\n"); #156789
    print("@removed\n"); #234
    @a = 1..9;
    @b = qw (- 9 8 7 6 5 4 3 2 1 -);
    splice(@a,1,3,@b); # 從 index 1 切掉三個之後加上 b 串列
    print("@a\n"); #1-987654321-56789
    @a = 1..9;
    splice(@a,1,0,@b); # 從 index 1 加上 b 串列, 完全不切掉任何值
    print("@a\n"); #1-987654321-23456789
    
    d:\>test.pl
    1
    1 5 6 7 8 9
    2 3 4
    1 - 9 8 7 6 5 4 3 2 1 - 5 6 7 8 9
    1 - 9 8 7 6 5 4 3 2 1 - 2 3 4 5 6 7 8 9
    
  30. print 的時候用 \@ 來跳脫串列的 @
    @yahoo = qw { yahoo hohoho };
    print("yahoo:@yahoo\n");
    print("mail:test@yahoo.com\n"); #@沒跳脫, 會換成串列內容
    print("mail:test\@yahoo.com\n"); #@跳脫了, 不會換成串列內容
    
    d:\>test.pl
    yahoo:yahoo hohoho
    mail:testyahoo hohoho.com
    mail:test@yahoo.com
    
  31. 串列可以當成陣列用
    @names = qw (a b c);
    print("index 0:$names[0]\n"); #a
    print("index 1:$names[1]\n"); #b
    print("index 2:$names[2]\n"); #c
    
    d:\>test.pl
    index 0:a
    index 1:b
    index 2:c
    
  32. 如果緊接著串列變數要印[index]的字串, 串列變數就要別處理
    @names = qw (a b c);
    print("index 0:${names[0]}[0]\n"); #用 {} 把變數圈起來
    print("index 1:$names[1]"."[1]\n"); #用 . 把字串分開
    print("index 2:$names[2]\[2\]\n"); #用 \ 跳脫 [ 與 ]
    
    d:\>test.pl
    index 0:a[0]
    index 1:b[1]
    index 2:c[2]
    
  33. foreach iterate 串列
    @names = qw (a b c);
    foreach $name (@names) {
     print("$name\n");
    }
    
    d:\>test.pl
    a
    b
    c
    
  34. foreach 裡面宣告的變數不會影響外部的變數
    $name = "hello";
    @names = qw (a b c);
    foreach $name (@names) {
     print("$name\n");
    }
    print("$name\n"); #hello
    
    d:\>test.pl
    a
    b
    c
    hello
    
  35. 預設變數 $_, 比方說在 foreach 的時候沒宣告變數就可以使用 $_
    foreach (qw / a b c /) {
     print("$_\n");
    }
    
    d:\>test.pl
    a
    b
    c
    
  36. reverse 把串列反過來
    @a = (1,2,3,4,5);
    print("a:@a\n");
    print("reverse a:".reverse(@a)."\n");
    
    d:\>test.pl
    a:1 2 3 4 5
    reverse a:54321
    
  37. 用 each iterate 串列, each 會一次回傳 index 與 value.
    @a = (1,2,3,4,5);
    while (($index,$value) = each(@a)) {
     print("index:$index, value:$value\n");
    }
    
    d:\>test.pl
    index:0, value:1
    index:1, value:2
    index:2, value:3
    index:3, value:4
    index:4, value:5
    
  38. 當進行字串的運算時, 就得到字串的結果. 當執行數字的計算時, 就得到數字的結果. 是字串還是數字是由運算符號決定.
    print(3*3 ."\n");
    print(3x3 ."\n");
    @a = qw{1 100 3 4 5}; #長度5
    print(3*@a ."\n"); #3*5=15
    print(3x@a ."\n");
    
    C:\Users\isaac>test.pl
    9
    333
    15
    33333
    
  39. 在字串的運算時, 串列會印出字串. 在數字的運算時, 串列會印出個數.
    @a = qw {e f d c b a};
    print(2*@a."\n"); #@a 是數字5, 印出 10 (2*5=10)
    print(2x@a."\n"); #@a 是數字5, 印出 22222
    print(sort(@a)); #印出排序過的字串
    
    C:\Users\isaac>test.pl
    12
    222222
    abcdef
    
  40. 運算串列的時候會印出串列, 但有時候運算串列的時候需要印出串列的 size. 這時候要用 scalar 這個假函式讓它變串列的 size
    @list = qw /a b c/;
    print("list:",@list,", size:",scalar @list);
    
    d:\>test.pl
    list:abc, size:3
    
  41. 可以在 console 多行資料給串列, 在 windows 下按 Ctrl+Z 結束, 在 Linux 下按 Ctrl+D 結束
    @commands = <STDIN>;
    print("commands:",@commands);
    
  42. d:\>test.pl
    a
    b
    c
    d
    e
    ^Z
    commands:a
    b
    c
    d
    e
    




  43. STDIN 輸入資料進串列, 每一行都會加上換行符號, 這不一定是我們要的, 可以用 chomp 去掉換行符號
    @commands = <STDIN>;
    chomp(@commands);
    print("commands:",@commands);
    

  44. d:\>test.pl
    a
    b
    c
    d
    e
    ^Z
    commands:abcde
    




  45. 可以簡化寫法
    chomp(@commands = <STDIN>);
    print("commands:",@commands);
    

  46. d:\>test.pl
    a
    b
    c
    d
    e
    ^Z
    commands:abcde
    




  47. 定義副常式 subroutine, 呼叫的方式是用 &副常式名稱 來呼叫.
    &hellosubroutine;
    
    sub hellosubroutine {
     print("hello subroutine");
    }
    
    d:\>test.pl
    hello subroutine
    
  48. subroutine 存取的變數都是全域變數
    &changeto5;
    print($n,"\n");
    &changeto10;
    print($n,"\n");
    
    sub changeto5 {
     $n = 5;
    }
    
    sub changeto10 {
     $n = 10;
    }
    
    d:\>test.pl
    5
    10
    
  49. subroutine 的最後一行計算就是回傳值
    print(&changeto5,"\n");
    print(&print,"\n");
    print(&add1ToN,"\n");
    print($n,"\n");
    
    sub add1ToN {
     $n + 1;
    }
    
    sub changeto5 {
     $n = 5;
    }
    
    sub print {
     print("");
    }
    
    d:\>test.pl
    5
    1
    6
    5
    
  50. subroutine 加參數
    sub test {
     print("arg[0]:$_[0]\n");
     print("arg[1]:$_[1]\n");
     print("arg[2]:$_[2]\n");
     print("arg[3]:$_[3]\n");
    }
    
    print("======3 args============\n");
    &test(1,2,3);
    print("======4 args============\n");
    &test(1,2,3,4);
    
    d:\>test.pl
    ======3 args============
    arg[0]:1
    arg[1]:2
    arg[2]:3
    arg[3]:
    ======4 args============
    arg[0]:1
    arg[1]:2
    arg[2]:3
    arg[3]:4
    
  51. 參數傳入 subroutine 後會存在 @_ 這個預設串列
    sub test {
     print("@_");
    }
    
    &test(1,2,3,4,5);
    
    d:\>test.pl
    1 2 3 4 5
    
  52. 用 my 可以宣告 subroutine 裡的區域變數
    sub test {
     $a = "a";
     my $b = "qq";
    }
    
    &test;
    print("a:$a\n");
    if (!defined($b)) {
     print("b is undef");
    }
    
    d:\>test.pl
    a:a
    b is undef
    
    sub max {
     my $max = shift @_;
     for (@_) {
      if ($max < $_) {
       $max = $_;  
      }
     }
     $max; #return
    }
    
    print(&max(1,2,3,4,5),"\n");
    if (!defined($max)) {
     print("\$max is undef");
    }
    
    d:\>test.pl
    5
    $max is undef
    
  53. 一個 subroutine 中本來就有一個變數, 又透過 my 宣告區域變數, subroutine 在 my 宣告後, 會以 my 宣告的變數值為主, 但又不影響原本的全域變數值
    sub max {
     $max = 333;
     my $max = shift @_;
     for (@_) {
      if ($max < $_) {
       $max = $_;  
      }
     }
     print($max,"\n"); #max=5
     $max; #return 5
    }
    
    print(&max(1,2,3,4,5),"\n");
    print($max); #max=333
    
    d:\>test.pl
    5
    5
    333
    
  54. 用 my 一次宣告多個變數來接外來的參數
    sub max {
     my($a,$b,$c,$d) = @_;
     print("a:$a,b:$b,c:$c,d:$d\n");
    }
    
    &max(1,2,3); 
    &max(1,2); 
    
    d:\>test.pl
    a:1,b:2,c:3,d:
    a:1,b:2,c:,d:
    
    
  55. 檢查陣列長度是否符合預期
    sub max {
     if (@_ != 2) {
      print("argument size should be 2\n");
     }
     my($a,$b) = @_;
     if ($a > $b) { 
      $a;
     } else {
      $b;
     }
    }
    
    print("max:",&max(1,2,3));
    
    D:\>test.pl
    argument size should be 2
    max:2
    
  56. use strict 強迫程式碼用比較好的方式撰寫 原本的範例
    sub test {
     foreach $qq (qw /a b c/) {
      print("$qq\n");
     }
    }
    
    $qq = 5;
    &test;
    print("qq:",$qq);
    
    d:\>test.pl
    a
    b
    c
    qq:5
    
    加上 use strict 之後
    use strict;
    sub test {
     foreach $qq (qw /a b c/) {
      print("$qq\n");
     }
    }
    
    $qq = 5;
    &test;
    print("qq:",$qq);
    
    d:\>test.pl
    Global symbol "$qq" requires explicit package name at D:\test.pl line 3.
    Global symbol "$qq" requires explicit package name at D:\test.pl line 4.
    Global symbol "$qq" requires explicit package name at D:\test.pl line 8.
    Global symbol "$qq" requires explicit package name at D:\test.pl line 10.
    Execution of D:\test.pl aborted due to compilation errors.
    
  57. return 回傳值
    原本 subroutine 的最後一行程式就是該 subroutine 的回傳值, 不過使用 return 就可以在最後一行之前回傳
    sub indexOf {
        my($keyword,@texts) = @_;
        foreach (0...$#texts) {
            if ($keyword eq $texts[$_]) {
                return $_;
            }
        }
        -1;
    }
    
    print(&indexOf("test",qw/ ab r ewr /),"\n");
    print(&indexOf("test",qw/ ab r ewr test/),"\n");
    
    d:\>test.pl
    -1
    3
    
  58. 當呼叫 subroutine 時需要用 & 來呼叫, 這是透過 & 來告訴 perl 這是一個 subroutine. 不過如果呼叫的時候有加參數,讓 perl 知道這是個 subroutine, 就不需要 & 了.
    sub say {
        print("say:",@_);
    }
    
    say("hello");
    
    d:\>test.pl
    say:hello
    
  59. 但是如果 subroutine 的名稱跟 perl 預設的 function 同名, 那還是需要透過 & 來告訴 perl 這是 subroutine 而不是預設的 function.
    sub print {
        print("print:",@_);
    }
    print("hello\n");
    &print("hello");
    
    d:\>test.pl
    hello
    print:hello
    
  60. 使用 my 宣告的區域變數在 subroutine 結束後值就不在了, 使用 state 宣告的話, 變數的狀態會記在 subroutine 中. 不過要宣告 use 5.010 才可以使用這個功能.
    use 5.010;
    
    sub test {
        my $localn = 0;
        $localn = $localn+1;
        print("test.localn:",$localn,"\n");
        
        state $n = 0;
        $n = $n+1;
        print("test.n:",$n,"\n");
    }
    
    sub test2 {
        state $n = 0;
        $n = $n+1;
        print("test2.n:",$n,"\n");
    }
    
    &test;
    &test;
    &test2;
    
    d:\>test.pl
    test.localn:1
    test.n:1
    test.localn:1
    test.n:2
    test2.n:1
    
    use 5.010;
    
    sub append {
        state @list;
        foreach (@_) {
            push(@list,$_);
        }
        print("list:",@list,"\n");
    }
    
    &append(qw/a b c/);
    &append(qw/1 2 3/);
    &append(qw/Q R T/);
    
    d:\>test.pl
    list:abc
    list:abc123
    list:abc123QRT
    
  61. console 輸入
    $line = <STDIN>;
    chomp($line);
    print($line);
    
    d:\>test.pl
    test
    test
    
    while (defined($line = <STDIN>)) {
        print($line);
    }
    
    d:\>test.pl
    test
    test
    qq
    qq
    BB
    BB
    ^Z
    
    d:\>
    
    while(<STDIN>) {
        print($_);
    }
    
    d:\>test.pl
    test
    test
    qq
    qq
    bb
    bb
    QQ
    QQ
    ^Z
    
    
    foreach (<STDIN>) {
        print($_,"\n");
    }
    
    d:\>test.pl
    a
    b
    c
    ^Z
    a
    
    b
    
    c
    
    這裡值得說明的是: perl 在 while 迴圈中使用 <STDIN> 做了特別處理, 使用 while (<STDIN>) 的效果會變這樣
    while (defined($_ = <STDIN>)) {
        print($_);
    }
    
    d:\>test.pl
    test
    test
    bb
    bb
    ^Z
    
    不過使用 foreach 則會把 STDIN 的結果全都讀進來才用 foreach iterate.
    這代表著如果 STDIN 的 input 量很大, 使用 while 沒關係因為每次換行都會輸出一次.
    使用 foreach 來讀大資料的話有可能一次佔用很多記憶體.
  62. 在程式中使用 while (<>) 可以讀取開啟程式時參數指定的檔案, 或者用 - 來當成標準輸入
    test2.txt
    {"test2":"test2","a": 1, "b": [1, 2, 3, 4, 5, 6]}
    
    julie.txt
    2.59,2.11,2:11,2:23,3-10,2-23,3:10,3.21,3-21
    
    test.pl
    while (<>) {
        print("print:$_\n");
    }
    
    d:\>test.pl test2.txt julie.txt
    print:{"test2":"test2","a": 1, "b": [1, 2, 3, 4, 5, 6]}
    print:2.59,2.11,2:11,2:23,3-10,2-23,3:10,3.21,3-21
    
    在參數指定 - 可以加上 STDIN 的效果
    test.pl
    while (<>) {
        print("print:$_\n");
    }
    
    d:\>test.pl test2.txt - julie.txt
    print:{"test2":"test2","a": 1, "b": [1, 2, 3, 4, 5, 6]}
    qq
    print:qq
    
    bb
    print:bb
    
    ^Z
    print:2.59,2.11,2:11,2:23,3-10,2-23,3:10,3.21,3-21
    
    看到 - 的處理都會多一個換行, 可以用 chomp 去掉.
    另外 perl 鼓勵我們少打字, 呼叫 function 的時候不用加括號也可以
    while (<>) {
        chomp;
        print "print:$_\n";
    }
    
    d:\>test.pl test2.txt - julie.txt
    print:{"test2":"test2","a": 1, "b": [1, 2, 3, 4, 5, 6]}
    testt
    print:testt
    ^Z
    print:2.59,2.11,2:11,2:23,3-10,2-23,3:10,3.21,3-21
    
    
    如果沒指定參數, <> 就會從 STDIN 讀取輸入
    while (<>) {
        chomp;
        print "print:$_\n";
    }
    
    d:\>test.pl
    a
    print:a
    b
    print:b
    c
    print:c
    ^Z
    
    
  63. while(<>) 其實是處理 @ARGV, @ARGV 是 perl 的特殊陣列, 裡面會放起動程式的參數, 進入程式後可以像一般陣列一樣使用
    foreach (@ARGV) {
        print("arg:$_\n");
    }
    
    d:\>test.pl a b c
    arg:a
    arg:b
    arg:c
    
    
    @ARGV = qw/a b c/;
    foreach (@ARGV) {
        print("arg:$_\n");
    }
    
    d:\>test.pl d d d
    arg:a
    arg:b
    arg:c
    
  64. print <> 作出 linux 下 cat 的效果 data1.txt
    a
    b
    c
    d
    e
    
    data2.txt
    d
    d
    c
    b
    a
    e
    
    執行 cat
    [root@Platform-151-ninja Isaac]# cat data1.txt data2.txt 
    a
    b
    c
    d
    e
    d
    d
    c
    b
    a
    e
    
    執行 perl
    print <>;
    
    在 linux 執行
    [root@Isaac]# perl test.pl data1.txt data2.txt 
    a
    b
    c
    d
    e
    d
    d
    c
    b
    a
    e
    
    在 windows 執行結果跟在 Linux 執行不太一樣
    d:\>test.pl data1.txt data2.txt
    a
    b
    c
    d
    ed
    d
    c
    b
    a
    e
    
  65. 待續...

Lessons Learned While Benchmarking vLLM with GPU

Recently, I benchmarked vLLM on a GPU to better understand how much throughput can realistically be expected in an LLM serving setup. One ...