fluentd, couchbase 3.0.2, ubuntu 14.04

1. install base
  sudo apt-get install build-essential
  sudo apt-get install ruby-dev

2. fluentd install

  http://docs.fluentd.org/articles/install-by-deb
  curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

3. couchbase install
  http://www.couchbase.com/nosql-databases/downloads
  wget http://packages.couchbase.com/releases/3.0.1/couchbase-server-community_3.0.1-ubuntu12.04_amd64.deb
  sudo dpkg -i couchbase-server-community_3.0.1-ubuntu12.04_amd64.deb

4. couchbase plugin

  wget http://packages.couchbase.com/clients/c/couchbase-csdk-setup
  perl couchbase-csdk-setup
  gem install couchbase
  sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-couchbase
 
5. couchbase rb file
vi /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-couchbase-0.0.2/lib/fluent/plugin/out_couchbase.rb

fluentd couchdb elasticsearch kibana

elasticsearch : 9200

couchdb : 5984

1. elasticsearch install

sudo apt-get install openjdk-7-jre-headless -y

sudo wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -

Add the following line to /etc/apt/sources.list

deb http://packages.elasticsearch.org/elasticsearch/1.5/debian stable main

sudo apt-get update

sudo apt-get install elasticsearch

sudo update-rc.d elasticsearch defaults 95 10

sudo /etc/init.d/elasticsearch start

2. fluentd install

curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-lucid.sh | sh

sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

3. fluentd 성능 개선

멀티 프로세스

http://docs.fluentd.org/articles/in_multiprocess

루비 메모리 문제

https://github.com/kzk/jemalloc-rb

4. elastic search couchdb 연결

- log 생성

curl -XPUT 'http://localhost:9200/news/'

cd /usr/share/elasticsearch/

./bin/plugin -install elasticsearch/elasticsearch-river-couchdb/1.2.0

curl -XPUT localhost:9200/news/news/_mapping -d '{

    "news" : {

         "numeric_detection" : false,

          "properties" : {

               "creation" : {"format":"date_time_no_millis","type" : "date"}

        }

    }

}'

curl -XPUT localhost:9200/news/news/_mapping -d '{

    "news" : {

         "numeric_detection" : false,

          "properties" : {

               "creation" : {"type" : "date"}

        }

    }

}'

curl -XPUT 'localhost:9200/_river/news/_meta' -d '{

     "type" : "couchdb",

     "couchdb" : {

          "host" : "122.36.208.82",

          "port" : 8888,

          "db" : "news",

          "filter" : null

     },

     "index" : {

          "index" : "news",

          "type" : "news",

          "bulk_size" : "100",

          "bulk_timeout" : "10ms"

     }

}'

- river 삭제

curl -XDELETE 'http://localhost:9200/_river/news/'

- log 삭제

curl -XDELETE 'http://localhost:9200/news/'

- mapping 삭제

curl -XDELETE 'http://localhost:9200/news/news/_mapping'

5. fluentd + geoip

sudo apt-get install libgeoip-dev

sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-geoip

<match low.lighttpd.log>

     type copy

     <store>

          type    geoip

          geoip_lookup_key  host

     <record>

          city  ${city['host']}

          lat   ${latitude['host']}

          lon   ${longitude['host']}

     </record>

     remove_tag_prefix low.

     tag     couch.${tag}

     </store>

</match>

------------------------------------------------------------------------------------------------------------------------

vi /etc/td-agent/td-agent.conf

<match td.*.*>

  type tdlog

  apikey YOUR_API_KEY

  auto_create_table

  buffer_type file

  buffer_path /var/log/td-agent/buffer/td

</match>

<match couch.**>

  type couch

  database log

  # following attibutes are optional

  host localhost                #default:localhost

  port 5984                   #default:5984

  protocol http               #default:http

  update_docs false            #default:false

  doc_key_field nil         #default:nil

  doc_key_jsonpath nil #default:nil

  refresh_view_index list  #default:nil

  # for Basic Authentication (optional)

  user admin

  password manson

  # Other buffer configurations here

        buffer_type file

        buffer_path /var/fluentd/buffer

</match>

## match tag=debug.** and dump to console

#<match debug.**>

#  type stdout

#</match>

####

## Source descriptions:

##

## built-in TCP input

## @see http://docs.fluentd.org/articles/in_forward

#<source>

#  type forward

#</source>

<match low.lighttpd.log>

  type copy

  <store>

    type    geoip

    geoip_lookup_key  host

    <record>

      city  ${city['host']}

      lat   ${latitude['host']}

      lon   ${longitude['host']}

    </record>

    remove_tag_prefix low.

    tag     nosql.${tag}

  </store>

</match>

<match nosql.**>

        type copy

        <store>

                type arango

                collection log

                buffer_type memory

                buffer_chunk_limit 256m

                buffer_queue_limit 128

"/etc/td-agent/td-agent.conf" 230L, 4801C                                                                                     1,1           Top

ubuntu 14.04 R install

very easy~ $ sudo apt-get install r-base r-base-dev

Centos에 yum으로 Git설치하기

$ yum install git
설치가 안되면

1. yum 쌧팅
/etc/yum.repos.d/epel.repo
위 파일을 만들고 아래와 같이 작성한다.

[epel]
name=Extra Packages for Enterprise Linux 5 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/5/$basearch
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 5 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/5/$basearch/debug
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-debug-5&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 5 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/5/SRPMS
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=epel-source-5&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
gpgcheck=1


2. rpm샛팅
$ cd /etc/pki/rpm-gpg/
위 폴더에 가서 RPM-GPG-KEY-EPEL 이 파일이 있는지 확인한다.
없다면 아래와 같이 작성하고 있다면 넘어가자 3번으로 넘어가자
$ wget http://download.fedora.redhat.com/pub/epel/RPM-GPG-KEY-EPEL
$ rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL

3. 설치
$ yum install git git-daemon

fluentd couchdb elasticsearch kibana ubuntu 14.04

- port -
elasticsearch : 9200
couchdb : 5984


1. elasticsearch install

$ sudo apt-get install openjdk-7-jre-headless -y
$ sudo wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -


$ sudo vi /etc/apt/sources.list
  -> deb http://packages.elasticsearch.org/elasticsearch/1.3/debian stable main

$ sudo apt-get update
$ sudo apt-get install elasticsearch
$ sudo update-rc.d elasticsearch defaults 95 10
$ sudo /etc/init.d/elasticsearch start


2. fluentd install
$ curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-lucid.sh | sh
$ sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch




3. fluentd power up
- multiprocess
  http://docs.fluentd.org/articles/in_multiprocess
- ruby memory fix
  https://github.com/kzk/jemalloc-rb

4. elasticsearch + couchdb

$ curl -XPUT 'http://localhost:9200/news/'
$ cd /usr/share/elasticsearch/
$ ./bin/plugin -install elasticsearch/elasticsearch-river-couchdb/1.2.0

 - date set time format, it is options
$ curl -XPUT localhost:9200/news/news/_mapping -d '{
    "news" : {
         "numeric_detection" : false,
          "properties" : {
               "creation" : {"format":"date_time_no_millis","type" : "date"}
        }
    }
}'
 - add river (it's connect couchdb)
$ curl -XPUT 'localhost:9200/_river/news/_meta' -d '{
     "type" : "couchdb",
     "couchdb" : {
          "host" : "localhost",
          "port" : 8888,
          "db" : "news",
          "filter" : null
     },
     "index" : {
          "index" : "news",
          "type" : "news",
          "bulk_size" : "100",
          "bulk_timeout" : "10ms"
     }
}'

- river delete
curl -XDELETE 'http://localhost:9200/_river/news/'

- news delete
curl -XDELETE 'http://localhost:9200/news/'


 - option
5. fluentd + geoip
$ sudo apt-get install libgeoip-dev
$ sudo /usr/lib/fluent/ruby/bin/fluent-gem install fluent-plugin-geoip

<match low.lighttpd.log>
     type copy
     <store>
          type    geoip
          geoip_lookup_key  host
     <record>
          city  ${city['host']}
          lat   ${latitude['host']}
          lon   ${longitude['host']}
     </record>
     remove_tag_prefix low.
     tag     couch.${tag}
     </store>
</match>

6. kibana download
http://www.elasticsearch.org/overview/kibana/

vi config.js
elasticsearch: "http://127.0.0.1:9200"

couchbase 3.0 [php client 1.2.1, libcouchbase-2.3.2] install from ubuntu 14.04

1. Download lib

  Download  URL : couchbase-1.2.1.zip

  Download URL : libcouchbase-2.3.2

2. unzip and upload your server

3. install libcouchbase 2.3.2

  $sudo apt-get install libev4

  $sudo dpkg -i ./libcouchbase2-core_2.3.2_amd64.deb

  $sudo dpkg -i ./libcouchbase-dev_2.3.2_amd64.deb 

  $sudo dpkg -i ./libcouchbase-dbg_2.3.2_amd64.deb 

  $sudo dpkg -i ./libcouchbase2-bin_2.3.2_amd64.deb 

  $sudo dpkg -i ./libcouchbase2-libev_2.3.2_amd64.deb 

  $sudo dpkg -i ./libcouchbase2-libevent_2.3.2_amd64.deb 

4. install couchbase-1.2.1
  $sudo apt-get install php5-dev
  $phpize
  $./configure
  $make
  $sudo make install

5. php setting
  $vi /etc/php5/cgi/conf.d/couchbase.ini
  extension=couchbase.so

  $vi /etc/php5/cli/conf.d/couchbase.ini
  extension=couchbase.so

CIFilter for iOS 5.0

CIColorControls

Adjusts saturation, brightness, and contrast values.

Parameters

inputImage

A CIImage class whose display name is Image.

inputSaturation

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Saturation.

Default value: 1.00 Minimum: 0.00 Maximum: 2.00 Slider minimum: 0.00 Slider maximum: 2.00 Identity: 1.00

inputBrightness

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Brightness.

Default value: 0.00 Minimum: -1.00 Maximum: 1.00 Slider minimum: -1.00 Slider maximum: 1.00 Identity: 0.00

inputContrast

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Contrast.

Default value: 1.00 Minimum: 0.00 Maximum: 4.00 Slider minimum: 0.00 Slider maximum: 4.00 Identity: 1.00

Discussion

To calculate saturation, this filter linearly interpolates between a grayscale image (saturation = 0.0) and the original image (saturation = 1.0). The filter supports extrapolation: For values large than 1.0, it increases saturation.

To calculate contrast, this filter uses the following formula:

(color.rgb - vec3(0.5)) * contrast + vec3(0.5)

This filter calculates brightness by adding a bias value:

color.rgb + vec3(brightness)

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Color Controls

Figure 18  The result of using the CIColorControls filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIColorMatrix

Multiplies source color values and adds a bias factor to each color component.

Parameters

inputImage

A CIImage class whose display name is Image.

inputRVector

A CIVector class whose display name is Red Vector.

Default value: [1 0 0 0] Identity: [1 0 0 0]

inputGVector

A CIVector class whose display name is Green Vector.

Default value: [0 1 0 0] Identity: [0 1 0 0]

inputBVector

A CIVector class whose display name is Blue Vector.

Default value: [0 0 1 0] Identity: [0 0 1 0]

inputAVector

A CIVector class whose display name is Alpha Vector.

Default value: [0 0 0 1] Identity: [0 0 0 1]

inputBiasVector

A CIVector class whose display name is Bias Vector.

Default value: [0 0 0 0] Identity: [0 0 0 0]

Discussion

This filter performs a matrix multiplication, as follows, to transform the color vector:

s.r = dot(s, redVector)

s.g = dot(s, greenVector)

s.b = dot(s, blueVector)

s.a = dot(s, alphaVector)

s = s + bias

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Color Matrix

Figure 23  The result of using the CIColorMatrix filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CICrop

Applies a crop to an image.

Parameters

inputImage

A CIImage class whose display name is Image.

inputRectangle

A CIVector class whose attribute type is CIAttributeTypeRectangle and whose display name is Rectangle.

Default value: [0 0 300 300] Identity: (null)

Discussion

The size and shape of the cropped image depend on the rectangle you specify.

Member of

CICategoryBuiltIn, CICategoryStillImage, CICategoryVideo, CICategoryGeometryAdjustment

Localized Display Name

Crop

Figure 29  The result of using the CICrop filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIExposureAdjust

Adjusts the exposure setting for an image similar to the way you control exposure for a camera when you change the F-stop.

Parameters

inputImage

A CIImage class whose display name is Image.

inputEV

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is EV.

Default value: 0.50 Minimum: 0.00 Maximum: 0.00 Slider minimum: -10.00 Slider maximum: 10.00 Identity: 0.00

Discussion

This filter multiplies the color values, as follows, to simulate exposure change by the specified F-stops:

s.rgb * pow(2.0, ev)

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Exposure Adjust

Figure 42  The result of using the CIExposureAdjust filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIGammaAdjust

Adjusts midtone brightness.

Parameters

inputImage

A CIImage class whose display name is Image.

inputPower

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Power.

Default value: 0.75 Minimum: 0.00 Maximum: 0.00 Slider minimum: 0.10 Slider maximum: 3.00 Identity: 1.00

Discussion

This filter is typically used to compensate for nonlinear effects of displays. Adjusting the gamma effectively changes the slope of the transition between black and white. It uses the following formula:

pow(s.rgb, vec3(power))

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Gamma Adjust

Figure 48  The result of using the CIGammaAdjust filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIHighlightShadowAdjust

Adjust the tonal mapping of an image while preserving spatial detail.

Parameters

inputImage

A CIImage class whose display name is Image.

inputHighlightAmount

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Highlight Amount.

Default value: 1.00 Minimum: 0.00 Maximum: 1.00 Slider minimum: 0.00 Slider maximum: 1.00 Identity: 1.00

inputShadowAmount

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Shadow Amount.

Default value: 0.00 Minimum: -1.00 Maximum: 1.00 Slider minimum: -1.00 Slider maximum: 1.00 Identity: 0.00

Member of

CICategoryBuiltIn, CICategoryStillImage, CICategoryVideo, CICategoryStylize

Localized Display Name

Highlight and Shadows

Figure 59  The result of using the CIHighlightShadowAdjust filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIHueAdjust

Changes the overall hue, or tint, of the source pixels.

Parameters

inputImage

A CIImage class whose display name is Image.

inputAngle

An NSNumber class whose attribute type is CIAttributeTypeAngle and whose display name is Angle.

Default value: 0.00 Minimum: 0.00 Maximum: 0.00 Slider minimum: -3.14 Slider maximum: 3.14 Identity: 0.00

Discussion

This filter essentially rotates the color cube around the neutral axis.

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Hue Adjust

Figure 61  The result of using the CIHueAdjust filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CISepiaTone

Maps the colors of an image to various shades of brown.

Parameters

inputImage

A CIImage class whose display name is Image.

inputIntensity

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Intensity.

Default value: 1.00 Minimum: 0.00 Maximum: 0.00 Slider minimum: 0.00 Slider maximum: 1.00 Identity: 0.00

Member of

CICategoryBuiltIn, CICategoryStillImage, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryVideo,CICategoryColorEffect

Localized Display Name

Sepia Tone

Figure 94  The result of using the CISepiaTone filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CITemperatureAndTint

Adapts the reference white point for an image.

Parameters

inputImage

A CIImage class whose display name is Image.

inputNeutral

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Neutral.

Default value: [6500, 0] Identity: [6500, 0]

inputTargetNeutral

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is TargetNeutral

Default value: [6500, 0] Identity: [6500, 0]

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Temperature and Tint

Figure 111  The result of using the CITemperatureAndTint filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIToneCurve

Adjusts tone response of the R, G, and B channels of an image.

Parameters

inputImage

A CIImage class whose display name is Image.

inputPoint0

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Point 1.

Default value: [0, 0] Identity: [0, 0]

inputPoint1

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Point 2l

Default value: [0.25, 0.25] Identity: [0.25, 0.25]

inputPoint2

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Point 3l

Default value: [0.5, 0.5] Identity: [0.5, 0.5]

inputPoint3

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Point 4

Default value: [0.75, 0.75] Identity: [0.75, 0.75]

inputPoint4

A CIVector class whose attribute type is CIAttributeTypeOffset and whose display name is Point 5

Default value: [1, 1] Identity: [1, 1]

Discussion

The input points are five x,y values that are interpolated using a spline curve. The curve is applied in a perceptual (gamma 2) version of the working space.

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Tone Curve

Figure 112  The result of using the CIToneCurve filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIVibrance

Adjusts the saturation of an image while keeping pleasing skin tones.

Parameters

inputImage

A CIImage class whose display name is Image.

inputAmount

An NSNumber class whose attribute type is CIAttributeTypeScalar and whose display name is Amount.

Default value: 0.00 Minimum: -1.00 Maximum: 1.00 Slider minimum: -1.00 Slider maximum: 1.00 Identity: 0.00

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

Vibrance

Figure 118  The result of using the CIVibrance filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.

CIWhitePointAdjust

Adjusts the reference white point for an image and maps all colors in the source using the new reference.

Parameters

inputImage

A CIImage class whose display name is Image.

inputColor

A CIColor class whose display name is Color.

Member of

CICategoryBuiltIn, CICategoryNonSquarePixels, CICategoryInterlaced, CICategoryStillImage, CICategoryVideo,CICategoryColorAdjustment

Localized Display Name

White Point Adjust

Figure 120  The result of using the CIWhitePointAdjust filter

Availability

• Available in Mac OS X v10.4 and later and in iOS 5.0 and later.