linux运维学习手册1

linux运维学习阶段:
1、系统安装: 学习常用linux操作系统安装(centos,ubuntu,debian,archlinux,lfs等),有时间都建议学习下lfs
2、常用命令学习: ls cat vi top uptime vmstate taif cut sed find 等常用命令,知其然,更要知其所以然。
3、常用服务软件安装: nginx apache redis mysql php tomcat 等服务环境安装
4、深入学习各类服务配置文件,shell脚本编写等
5、各类服务组合架构

routeros ipsec with centos host to host 加密

routeros 设置

/ip ipsec peer add address=xx.xx.xx.xx/32 port=500 auth-method=pre-shared-key secret=”xxxxx” generate-policy=no exchange-mode=main
send-initial-contact=yes nat-traversal=no my-id-user-fqdn=”” proposal-check=obey hash-algorithm=sha1 enc-algorithm=3des
dh-group=modp1024 lifetime=1d lifebytes=0 dpd-interval=disable-dpd dpd-maximum-failures=5
/ip ipsec proposal add name=”default” auth-algorithms=sha1 enc-algorithms=3des lifetime=0s pfs-group=none
/ip ipsec policy add src-address=xx.xx.xx.xx/32 src-port=any dst-address=xx.xx.xx.xx/32 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=ah-esp tunnel=no sa-src-address=xx.xx.xx.xx sa-dst-address=xx.xx.xx.xx proposal=default priority=0

centos 安装

ipsec-tools
配置文件

/etc/racoon/racoon.conf

path include “/etc/racoon”;
path pre_shared_key “/etc/racoon/psk.txt”;
path certificate “/etc/racoon/certs”;
path script “/etc/racoon/scripts”;

remote anonymous {
exchange_mode aggressive, main, base;
mode_cfg on;
proposal_check obey;
nat_traversal on;
generate_policy unique;
ike_frag on;
passive on;
dpd_delay 30;

proposal {
    lifetime time 28800 sec;
    encryption_algorithm 3des;
    hash_algorithm md5;
    authentication_method xauth_psk_server;
    dh_group 2;
}

}

sainfo anonymous {
encryption_algorithm aes, 3des, blowfish;
authentication_algorithm hmac_sha1, hmac_md5;
compression_algorithm deflate;
}

include “/etc/racoon/xx.xx.xx.xx.conf”;

/etc/racoon/xxxxx.conf

remote xx.xx.xx.xx
{
exchange_mode main, aggressive;
my_identifier address;
proposal {
encryption_algorithm 3des;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
}

/etc/racoon/psk.txt

xx.xx.xx.xx password

[转]Using USB pass-through under libvirt and KVM

Using USB pass-through under libvirt and KVM

Virtualization solutions typically include a feature called USB pass-through: making a USB device attached to the host machine appear directly as a USB device attached to a virtual machine. KVM, the fully open-source virtualization solution for Linux, can do USB pass-through. It inherits this feature from QEMU, which KVM incorporates to provide system and device virtualization (KVM proper is focused on processor virtualization, and runs in kernel space; QEMU runs in user space).

But there’s a hitch: KVM is most often used via libvirt and virt-manager, which provide a virtualization management infrastructure and graphical user interface. libvirt has some support for USB pass-through; but virt-manager doesn’t support it at all. So here’s a guide to using the libvirt command shell to get to this feature. These instructions require somewhat recent versions of KVM (74 and up) and libvirt (0.4.4 and up).

You will need to hand-edit the XML documents that libvirt uses to describe virtual machine instances (domains, in libvirt terminology). You do this via virsh, the command shell included with libvirt. virsh is fairly spartan, though it does have a decent on-line help facility (accessed via the help command). Run virsh as root, and use the list command to see your VM instances and their names:

virsh # list –all

Id Name State

  • windowsxp shut off
    If list –all didn’t show any VM instances, you should create some using virt-manager.

You can view the domain definition XML with the dumpxml command:

virsh # dumpxml windowsxp

windowsxp


The edit command will open the domain definition XML in an editor:

virsh # edit windowsxp
Edit the XML as desired
Domain windowsxp XML configuration edited.
Update: I recently noticed that the edit command only appeared in libvirt version 0.4.5. Some current editions of popular Linux distributions still have an earlier version. In that case, you can edit the XML domain definition files under /etc/libvirt/qemu directly, but you will need to restart libvirtd afterwards.

We want to edit this XML to add a hostdev element as documented here on the libvirt site. But first, you need to work out the USB vendor and product IDs for the relevant device. You can discover these by attaching the device and running /sbin/lsusb on the host. For example, here’s the lsusb output on my laptop:

$ /sbin/lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 012: ID 0a5c:2110 Broadcom Corp. Bluetooth Controller
Bus 005 Device 003: ID 0483:2016 SGS Thomson Microelectronics Fingerprint Reader
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
The pairs of hexadecimal numbers on each line are the vendor and produce IDs. For example, the bluetooth controller has a vendor ID of 0a5c, and a product ID of 2110.

To forward that to my windowsxp VM, I edit the domain definition as follows:


windowsxp



</hostdev>



Note that it’s a good idea to prevent the host from using the device before attaching it to a VM. In this case, I made sure the bluetooth controller was not being used by removing the relevant kernel modules with rmmod. For USB storage devices, it’s sufficient to make sure that they are not mounted by the host kernel. And for custom USB devices, without drivers in the Linux kernel, it won’t try to do anything with them, so you’re usually fine.

There are a couple of limitations to this USB pass-through support. The first is that the change won’t take effect until you next start the VM. The second is that it only works if the USB device in question is connect at the time you start the VM. That’s quite a serious restriction. It’s often convenient, and sometimes necessary, to be able to connect a device to, and disconnect it from, a running machine. In a follow-up post, I’ll describe a way to avoid this limitation. It’s now up here.

But once it’s in operation, this USB pass-through support does work well. I’ve used it with a range of USB devices, with complete success.

www.photonvps.com 坑货

photonvps.com 的VPS 标称是SSD
结果检测一下

dd if=/dev/zero of=test.bin bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 30.1411 s, 34.8 MB/s

这个连HDD都不是

申请退款,客服老让你检测什么的,没意思,不就一个月么,坑得一B

nginx 配置文件解释

配置文件示例

常用配置: nginx.conf

配置区域层级

main
    events
    http
        server
            location
    stream
    mail

参数解析

core核心模块

user

Syntax: user user [group];
Default:    user nobody nobody;
Context:    main
定义运行nginx进程的用户和用户组

worker_processes

Syntax: worker_processes number | auto;
Default: ---
worker_processes 1;
Context:    main
设置nginx进程数量,一般设置成cpu数量*2
例: 4核心cpu
worker_processes 8
设置成auto则nginx自动根据系统确定数量

worker_cpu_affinity

Syntax: worker_cpu_affinity cpumask ...;
Default:    —
Context:    main
设置进程CPU绑定,有利于利用多核CPU资源
cpu数量 和 进程数量相关
例:
    4核心 4个nginx work
    worker_cpu_affinity 1000 0100 0010 0001

error_log

Syntax: error_log file | stderr | syslog:server=address[,parameter=value] | memory:size [debug | info | notice | warn | error | crit | alert | emerg];
Default:    error_log logs/error.log error;
Context:    main, http, mail, stream, server, location
设置错误日志文件file路径
stderr直接输出到屏幕
syslog输出到syslog日志系统
[debug | info | notice | warn | error | crit | alert | emerg] 记录错误等级

accept_mutex

指令格式  :   accept_mutex on | off;
默认值:    accept_mutex on;
使用区域:   events;
设置网络连接的序列化。 在Nginx服务器的多进程下,有可能出现惊群(Thundering herd problem)问题,
指的是当某一个时刻只有一个网络连接到来时,多个睡眠进程会被同时唤醒,但只有一个进程可以获得连接。
如果每次唤醒的进程数目太多,会影响一部分系统性能。。
为了解决这样的问题,Nginx配置中包含这样一条指令accept_mutex,
当其设置为开启时,将会对多个Nginx进程接受连接进行序列化,阻止多个进程对连接的争抢。
其语法结构为 accept_mutex on|off. 此指令默认为开启状态。
关于nginx解决惊群的方法可以参见http://blog.csdn.net/russell_tao/article/details/7204260

pid

Syntax: pid file;
Default:    pid nginx.pid;
Context:    main
设置nginx的pid文件路径

events {}

Syntax: events { ... }
Default:    —
Context:    main
设置events区域的参数

include

Syntax: include file | mask;
Default:    —
Context:    任何区域
包含、引用文件

multi_accept

示例  :   multi_accept on | off;
默认值:    multi_accept off;
区域  :   events
设置worker接受多个新连接,默认一个worker一次只接受一个新连接,multi_accept设置为on后能接受多个,有利于性能提升

use

 使用哪种网络模型(Connection processing methods)
 select 默认
 poll 默认
 kqueue BSD系统使用
 epoll linux 2.6以上
 /dev/poll|eventport  solaris unix系统使用
 linux使用 use epoll;

worker_connections

 Syntax:    worker_connections number;
 Default: —
 worker_connections 512;
 Context:   events

 设置nginx每个worker最大同时连接数
 与worker_processes一起限制了系统最大允许连接数worker_processes*worker_connections

worker_rlimit_nofile

Syntax: worker_rlimit_nofile number;
Default:    —
Context:    main

设置进程打开文件数量限制

http核心模块

default_type

Syntax: default_type mime-type;
Default:    
default_type text/plain;
Context:    http, server, location

设置默认文件类型mime-type

error_page

Syntax: error_page code ... [=[response]] uri;
Default:    —
Context:    http, server, location, if in location

设置如果出现指定的HTTP错误状态码,返回给客户端显示的对应uri地址。
error_page 404 /404.html;

msie_padding

Syntax: msie_padding on | off;
Default:    msie_padding on;
Context:    http, server, location

关闭或开启MSIE浏览器的msie_padding特性,
若启用选项,nginx会为response头部填满512字节,这样就阻止了相关浏览器会激活友好错误界面,
因此不会隐藏更多的错误信息。

server_tokens

Syntax: server_tokens on | off;
Default:    server_tokens on;
Context:    http, server, location

当打开server_tokens的时候,返回详细nginx版本信息

server_names_hash_bucket_size

Syntax: server_names_hash_bucket_size size;
Default:    server_names_hash_bucket_size 32|64|128;
Context:    http

保存服务器名字的hash表是由指令server_names_hash_max_size 和server_names_hash_bucket_size所控制的。
参数hash bucket size总是等于hash表的大小,并且是一路处理器缓存大小的倍数。在减少了在内存中的存取次数后,
使在处理器中加速查找hash表键值成为可能。如果hash bucket size等于一路处理器缓存的大小,那么在查找键的时候,
最坏的情况下在内存中查找的次数为2。第一次是确定存储单元的地址,第二次是在存储单元中查找键 值。
因此,如果Nginx给出需要增大hash max size 或 hash bucket size的提示,那么首要的是增大前一个参数的大小.

client_header_buffer_size

Syntax: client_header_buffer_size size;
Default:  client_header_buffer_size 1k;
Context:    http, server

用于设置客户端请求的Header头缓冲区大小,大部分情况1KB大小足够。
不能超过large_client_header_buffers缓冲区大小的设置

large_client_header_buffers

Syntax: large_client_header_buffers number size;
Default:    large_client_header_buffers 4 8k;
Context:    http, server

客户请求头缓冲大小
nginx默认会用client_header_buffer_size这个buffer来读取header值,
如果header过大,它会使用large_client_header_buffers来读取
如果设置过小而HTTP头/Cookie过大会报400错误nginx 400 bad request

client_max_body_size

Syntax: client_max_body_size size;
Default:    client_max_body_size 1m;
Context:    http, server, location

#设置客户端能够上传的文件大小,默认为1m

sendfile

Syntax: sendfile on | off;
Default:    sendfile off;
Context:    http, server, location, if in location

sendfile指令指定 nginx 是否调用sendfile 函数(zero copy 方式)来输出文件,
对于普通应用,设为on。
如果用来进行下载等应用磁盘IO重负载应用,可设置为off,以平衡磁盘与网络IO处理速度,降低系统uptime。

常用变量

rsyslog for mysql 表按月存放

$template StdSQLFormat,”insert into SystemEvents%$year%%$month% (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values (‘%msg%’, %syslogfacility%, ‘%fromhost-ip%’, %syslogpriority%, ‘%timereported:::date-mysql%’, ‘%timegenerated:::date-mysql%’, %iut%, ‘%syslogtag%’)”,SQL

$ModLoad ommysql.so
*.* :ommysql:127.0.0.1,rsyslogd,rsyslogd,xxxx;StdSQLFormat

Python 日志统计 一例[@]

@http://blog.aoath.com/archives/674.html
源文件:

file name: abc

218.79.251.215 - - [23/May/2006:08:57:44 +0800] "GET /fg172.exe HTTP/1.1" 206 2350253
220.178.150.3 - - [23/May/2006:08:57:40 +0800] "GET /fg172.exe HTTP/1.1" 200 2350253
59.42.2.185 - - [23/May/2006:08:57:52 +0800] "GET /fg172.exe HTTP/1.1" 200 2350253
219.140.190.130 - - [23/May/2006:08:57:59 +0800] "GET /fg172.exe HTTP/1.1" 200 2350253
221.228.143.52 - - [23/May/2006:08:58:08 +0800] "GET /fg172.exe HTTP/1.1" 206 719996
221.228.143.52 - - [23/May/2006:08:58:08 +0800] "GET /fg172.exe HTTP/1.1" 206 713242
221.228.143.52 - - [23/May/2006:08:58:09 +0800] "GET /fg172.exe HTTP/1.1" 206 1200250

示例:

#!/usr/bin/env python

sum = []
with open('abc', 'r') as file:
    for i in file.readlines():
        ip = [ i.split()[0]]
        if ip not in sum:
            sum.append(ip)
    print len(sum)

# End

输出结果:

5

## ip 去重,并统计 ip 数。

示例 2:( 优化版 )

#!/usr/bin/env python

a={}
with open('access_log', 'r') as file:
    for i in file.readlines():
        ip = i.split()[0]
        try:
            a[ '%s' % ip ] = a[ '%s' % ip ] + 1
        except:
            a[ '%s' % ip ] = 1
print len(a)

# End

## 速度是上一个版本的 7 倍之多 !

solr 主从配置

Master配置:

<requesthandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
<str name="confFiles">schema.xml,stopwords.txt,solrconfig.xml,synonyms.txt</str>
<str name="commitReserveDuration">00:00:10</str>
</lst>
</requesthandler>

Repeater配置:

<requesthandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
<str name="commitReserveDuration">00:00:10</str>
</lst>
<lst name="slave">
<str name="masterUrl">http://192.168.0.184:8080/masterapp/master/replication</str>
<str name="pollInterval">00:00:20</str>
<str name="compression">internal</str>
<str name="httpConnTimeout">5000</str>
<str name="httpReadTimeout">10000</str>
<str name="httpBasicAuthUser">username</str>
<str name="httpBasicAuthPassword">password</str>
</lst>
</requesthandler>

Slave配置:


<requesthandler name="/replication" class="solr.ReplicationHandler" >
<lst name="slave">
<str name="masterUrl">http://192.168.0.174:8080/repeaterapp/repeater/replication</str>
<str name="pollInterval">00:00:20</str>
<str name="compression">internal</str>
<str name="httpConnTimeout">5000</str>
<str name="httpReadTimeout">10000</str>
<str name="httpBasicAuthUser">username</str>
<str name="httpBasicAuthPassword">password</str>
</lst> </requesthandler>

excite.co.jp 翻译代码 PHP 示例

<?php
/**
* Created by IntelliJ IDEA.
* User: Administrator
* Date: 2014-12-02
* Time: 14:51
*/
set_time_limit(0);
include "Snoopy.class.php";
$snoopy = new Snoopy;
$submit_url = "http://www.excite.co.jp/world/chinese/";

$submit_vars["before"] = "日语";
$submit_vars["after"] = "";
$submit_vars["wb_lp"] = "JACH";
$submit_vars["big5"]="no";
$submit_vars["start"] = "%E7%BF%BB+%E8%A8%B3";
$snoopy->submit($submit_url,$submit_vars);
$result=$snoopy->getResults();
preg_match("/<textarea id=\"after\" cols=\"37\" rows=\"13\" name=\"after\">(.*)\<\/textarea\>/",$result,$match);
$match[1]=html_entity_decode($match[1]);
print_r($match);